Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 13:59:37.886006 17252 test_util.cc:268] Using random seed: 59140394
I20240502 13:59:37.901160 17252 ts_itest-base.cc:112] Starting cluster with:
I20240502 13:59:37.901225 17252 ts_itest-base.cc:113] --------------
I20240502 13:59:37.901242 17252 ts_itest-base.cc:114] 8 tablet servers
I20240502 13:59:37.901258 17252 ts_itest-base.cc:115] 3 replicas per TS
I20240502 13:59:37.901319 17252 ts_itest-base.cc:116] --------------
2024-05-02T13:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T13:59:37Z Disabled control of system clock
I20240502 13:59:37.909272 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46605
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:46605
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 13:59:37.996309 17264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:37.996471 17264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:37.996527 17264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:37.999112 17264 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:37.999168 17264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 13:59:37.999200 17264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:37.999234 17264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 13:59:37.999261 17264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 13:59:38.002257 17264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:46605
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.002872 17264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.003684 17264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 13:59:38.010294 17264 server_base.cc:1034] running on GCE node
W20240502 13:59:38.010854 17272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.011355 17269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.011747 17270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.012099 17264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.012840 17264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.013984 17264 hybrid_clock.cc:648] HybridClock initialized: now 1714658378013969 us; error 27 us; skew 500 ppm
I20240502 13:59:38.019987 17264 webserver.cc:460] Webserver started at http://127.16.217.62:41787/ using document root <none> and password file <none>
I20240502 13:59:38.020473 17264 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.020530 17264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.020697 17264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.021979 17264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.022390 17264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.025112 17264 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:38.026818 17278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.027622 17264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.027742 17264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.027819 17264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.040119 17264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.040606 17264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.040730 17264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.046262 17264 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:46605
I20240502 13:59:38.046283 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:46605 every 8 connection(s)
I20240502 13:59:38.046959 17264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 13:59:38.049187 17331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:38.053323 17331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: Bootstrap starting.
I20240502 13:59:38.053422 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17264
I20240502 13:59:38.053548 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 13:59:38.055652 17331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:38.056336 17331 log.cc:824] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:38.057891 17331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: No bootstrap required, opened a new log
I20240502 13:59:38.060081 17331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } }
I20240502 13:59:38.060223 17331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:38.060314 17331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae6c0b12f24436dabe8f1a0db52ad3f, State: Initialized, Role: FOLLOWER
I20240502 13:59:38.060680 17331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } }
I20240502 13:59:38.060784 17331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 13:59:38.060839 17331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 13:59:38.060933 17331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:38.061513 17331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } }
I20240502 13:59:38.061769 17331 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae6c0b12f24436dabe8f1a0db52ad3f; no voters:
I20240502 13:59:38.061975 17331 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 13:59:38.062043 17336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:38.062235 17336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [term 1 LEADER]: Becoming Leader. State: Replica: 9ae6c0b12f24436dabe8f1a0db52ad3f, State: Running, Role: LEADER
I20240502 13:59:38.062507 17336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } }
I20240502 13:59:38.062863 17331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [sys.catalog]: configured and running, proceeding with master startup.
I20240502 13:59:38.063858 17338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae6c0b12f24436dabe8f1a0db52ad3f. Latest consensus state: current_term: 1 leader_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } } }
I20240502 13:59:38.063879 17337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae6c0b12f24436dabe8f1a0db52ad3f" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 46605 } } }
I20240502 13:59:38.063983 17338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [sys.catalog]: This master's current role is: LEADER
I20240502 13:59:38.063983 17337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f [sys.catalog]: This master's current role is: LEADER
W20240502 13:59:38.066814 17351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 13:59:38.066879 17351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 13:59:38.067077 17345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 13:59:38.067824 17345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 13:59:38.071832 17345 catalog_manager.cc:1347] Generated new cluster ID: 72b8cbd72fdc4d48813d996533eb5d9d
I20240502 13:59:38.071887 17345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 13:59:38.087945 17345 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 13:59:38.088960 17345 catalog_manager.cc:1504] Loading token signing keys...
I20240502 13:59:38.103384 17345 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9ae6c0b12f24436dabe8f1a0db52ad3f: Generated new TSK 0
I20240502 13:59:38.103927 17345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 13:59:38.116618 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:38.219357 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.219521 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.219584 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.222090 17355 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.222157 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.222236 17355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20240502 13:59:38.225124 17355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.17355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.225762 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.226614 17355 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.233278 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.233839 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.234431 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.235028 17355 server_base.cc:1034] running on GCE node
I20240502 13:59:38.235299 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.235706 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.236837 17355 hybrid_clock.cc:648] HybridClock initialized: now 1714658378236819 us; error 33 us; skew 500 ppm
I20240502 13:59:38.238288 17355 webserver.cc:460] Webserver started at http://127.16.217.1:36265/ using document root <none> and password file <none>
I20240502 13:59:38.238718 17355 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.238768 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.238905 17355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.240170 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b36b507eb389473b8438e907237a63f9"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.240577 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b36b507eb389473b8438e907237a63f9"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.243166 17355 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.244870 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.245626 17355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.245716 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b36b507eb389473b8438e907237a63f9"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.245783 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.255671 17355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.256093 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.256215 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.256615 17355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.257310 17355 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.257354 17355 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.257401 17355 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.257431 17355 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.264397 17355 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:35657
I20240502 13:59:38.264434 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:35657 every 8 connection(s)
I20240502 13:59:38.265074 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 13:59:38.271906 17483 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:38.271950 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17355
I20240502 13:59:38.272046 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 13:59:38.272066 17483 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.272500 17483 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:38.273725 17294 ts_manager.cc:194] Registered new tserver with Master: b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:38.273885 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:38.275050 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:34143
W20240502 13:59:38.361799 17486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.361948 17486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.361985 17486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.364480 17486 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.364523 17486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.364578 17486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20240502 13:59:38.367493 17486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.17486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.368085 17486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.368934 17486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.375430 17494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.375875 17491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.376459 17492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.377192 17486 server_base.cc:1034] running on GCE node
I20240502 13:59:38.377528 17486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.377970 17486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.379112 17486 hybrid_clock.cc:648] HybridClock initialized: now 1714658378379101 us; error 23 us; skew 500 ppm
I20240502 13:59:38.380554 17486 webserver.cc:460] Webserver started at http://127.16.217.2:46447/ using document root <none> and password file <none>
I20240502 13:59:38.380961 17486 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.381014 17486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.381160 17486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.382386 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.382756 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.385347 17486 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:38.387004 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.387826 17486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.387938 17486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.388010 17486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.396340 17486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.396756 17486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.396878 17486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.397305 17486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.398007 17486 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.398054 17486 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.398106 17486 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.398144 17486 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.404976 17486 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:42667
I20240502 13:59:38.405001 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:42667 every 8 connection(s)
I20240502 13:59:38.405627 17486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 13:59:38.408390 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17486
I20240502 13:59:38.408478 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 13:59:38.410516 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:38.413084 17614 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:38.413254 17614 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.413651 17614 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:38.414407 17294 ts_manager.cc:194] Registered new tserver with Master: 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:38.414863 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:40407
W20240502 13:59:38.498698 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.498858 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.498919 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.501569 17617 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.501622 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.501708 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20240502 13:59:38.504542 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.505156 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.505971 17617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.512490 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.513053 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.513653 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.514199 17617 server_base.cc:1034] running on GCE node
I20240502 13:59:38.514523 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.515008 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.516149 17617 hybrid_clock.cc:648] HybridClock initialized: now 1714658378516137 us; error 27 us; skew 500 ppm
I20240502 13:59:38.517683 17617 webserver.cc:460] Webserver started at http://127.16.217.3:44821/ using document root <none> and password file <none>
I20240502 13:59:38.518127 17617 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.518196 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.518349 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.519572 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b0da919a0055405e8ef4dfd545e3bd23"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.519934 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b0da919a0055405e8ef4dfd545e3bd23"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.522519 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:38.524163 17631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.524957 17617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:38.525040 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b0da919a0055405e8ef4dfd545e3bd23"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.525108 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.539207 17617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.539625 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.539750 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.540129 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.540865 17617 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.540907 17617 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.540952 17617 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.540982 17617 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.548143 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:42979 every 8 connection(s)
I20240502 13:59:38.548152 17617 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:42979
I20240502 13:59:38.548918 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 13:59:38.555032 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17617
I20240502 13:59:38.555119 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 13:59:38.555593 17745 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:38.555775 17745 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.556202 17745 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:38.556995 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.4:0
--local_ip_for_outbound_sockets=127.16.217.4
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:38.557049 17294 ts_manager.cc:194] Registered new tserver with Master: b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:38.557544 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:37183
W20240502 13:59:38.645769 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.645922 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.645958 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.648650 17748 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.648702 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.648787 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.4
I20240502 13:59:38.651708 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.217.4
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.17748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.4
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.652352 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.653167 17748 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.659734 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.660250 17754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.660856 17753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.661456 17748 server_base.cc:1034] running on GCE node
I20240502 13:59:38.661705 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.662071 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.663198 17748 hybrid_clock.cc:648] HybridClock initialized: now 1714658378663193 us; error 25 us; skew 500 ppm
I20240502 13:59:38.664718 17748 webserver.cc:460] Webserver started at http://127.16.217.4:36837/ using document root <none> and password file <none>
I20240502 13:59:38.665128 17748 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.665187 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.665343 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.666618 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e7f378d8d4c745d888ed99454393bb85"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.666986 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e7f378d8d4c745d888ed99454393bb85"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.669746 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:38.671540 17762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.672338 17748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 13:59:38.672417 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e7f378d8d4c745d888ed99454393bb85"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.672485 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.687781 17748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.688261 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.688431 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.688841 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.689532 17748 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.689574 17748 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.689617 17748 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.689642 17748 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.696588 17748 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.4:40769
I20240502 13:59:38.696620 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.4:40769 every 8 connection(s)
I20240502 13:59:38.697283 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 13:59:38.701715 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17748
I20240502 13:59:38.701787 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 13:59:38.703807 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:0
--local_ip_for_outbound_sockets=127.16.217.5
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:38.704250 17876 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:38.704452 17876 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.704952 17876 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:38.705868 17294 ts_manager.cc:194] Registered new tserver with Master: e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:38.706319 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.4:55067
W20240502 13:59:38.793030 17879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.793179 17879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.793215 17879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.795621 17879 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.795668 17879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.795745 17879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 13:59:38.798804 17879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.17879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.799448 17879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.800252 17879 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.806631 17887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.807243 17885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.807828 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.808476 17879 server_base.cc:1034] running on GCE node
I20240502 13:59:38.808727 17879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.809095 17879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.810223 17879 hybrid_clock.cc:648] HybridClock initialized: now 1714658378810218 us; error 26 us; skew 500 ppm
I20240502 13:59:38.811652 17879 webserver.cc:460] Webserver started at http://127.16.217.5:41331/ using document root <none> and password file <none>
I20240502 13:59:38.812129 17879 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.812201 17879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.812444 17879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.813699 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3305e392697b48a8ad63900628ce4c27"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.814066 17879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3305e392697b48a8ad63900628ce4c27"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.816627 17879 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:38.818423 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.819211 17879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:38.819306 17879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3305e392697b48a8ad63900628ce4c27"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.819371 17879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.834259 17879 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.834698 17879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.834820 17879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.835206 17879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.835861 17879 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.835903 17879 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.835949 17879 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.835976 17879 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.843070 17879 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:35713
I20240502 13:59:38.843093 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:35713 every 8 connection(s)
I20240502 13:59:38.843715 17879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:38.848766 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 17879
I20240502 13:59:38.848860 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 13:59:38.850620 18007 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:38.850783 18007 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:38.850880 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:0
--local_ip_for_outbound_sockets=127.16.217.6
--webserver_interface=127.16.217.6
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:38.851210 18007 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:38.852038 17294 ts_manager.cc:194] Registered new tserver with Master: 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713)
I20240502 13:59:38.852577 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:50477
W20240502 13:59:38.938784 18010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:38.938925 18010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:38.938968 18010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:38.941426 18010 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:38.941474 18010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:38.941557 18010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 13:59:38.944506 18010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.18010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:38.945111 18010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:38.945904 18010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:38.952543 18015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.953130 18016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:38.953711 18018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:38.954319 18010 server_base.cc:1034] running on GCE node
I20240502 13:59:38.954573 18010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:38.954988 18010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:38.956121 18010 hybrid_clock.cc:648] HybridClock initialized: now 1714658378956105 us; error 26 us; skew 500 ppm
I20240502 13:59:38.957679 18010 webserver.cc:460] Webserver started at http://127.16.217.6:45669/ using document root <none> and password file <none>
I20240502 13:59:38.958124 18010 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:38.958189 18010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:38.958333 18010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:38.959535 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "73731152a18648aab3b1a720dabcc26b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.959903 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "73731152a18648aab3b1a720dabcc26b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.962404 18010 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:38.964068 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.964851 18010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:38.964951 18010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "73731152a18648aab3b1a720dabcc26b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:38.965015 18010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:38.987982 18010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:38.988469 18010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:38.988591 18010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:38.988986 18010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:38.989638 18010 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:38.989681 18010 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.989727 18010 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:38.989750 18010 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:38.996542 18010 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:40157
I20240502 13:59:38.996598 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:40157 every 8 connection(s)
I20240502 13:59:38.997242 18010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:39.004161 18138 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:39.004356 18138 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.004724 18138 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:39.005558 17294 ts_manager.cc:194] Registered new tserver with Master: 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157)
I20240502 13:59:39.005605 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 18010
I20240502 13:59:39.005693 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 13:59:39.006065 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:41995
I20240502 13:59:39.007565 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:0
--local_ip_for_outbound_sockets=127.16.217.7
--webserver_interface=127.16.217.7
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:39.100680 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.100837 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.100901 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.103368 18141 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.103420 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.103510 18141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 13:59:39.106413 18141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.18141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.106982 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.107844 18141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.114614 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.115151 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.115572 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.116183 18141 server_base.cc:1034] running on GCE node
I20240502 13:59:39.116477 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.116925 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.118074 18141 hybrid_clock.cc:648] HybridClock initialized: now 1714658379118052 us; error 38 us; skew 500 ppm
I20240502 13:59:39.119622 18141 webserver.cc:460] Webserver started at http://127.16.217.7:42705/ using document root <none> and password file <none>
I20240502 13:59:39.120043 18141 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.120098 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.120244 18141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.121559 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.121934 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.124487 18141 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 13:59:39.126246 18155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.126979 18141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 13:59:39.127069 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.127175 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.139603 18141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.140061 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.140187 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.140596 18141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.141249 18141 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.141292 18141 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.141336 18141 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.141363 18141 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.148258 18141 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:44123
I20240502 13:59:39.148357 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:44123 every 8 connection(s)
I20240502 13:59:39.152835 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 13:59:39.159431 18269 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:39.159575 18269 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.159919 18269 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:39.160663 17294 ts_manager.cc:194] Registered new tserver with Master: d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123)
I20240502 13:59:39.161106 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:58307
I20240502 13:59:39.161995 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 18141
I20240502 13:59:39.162086 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 13:59:39.163766 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.8:0
--local_ip_for_outbound_sockets=127.16.217.8
--webserver_interface=127.16.217.8
--webserver_port=0
--tserver_master_addrs=127.16.217.62:46605
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:39.251763 18272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:39.251924 18272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:39.251981 18272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:39.254472 18272 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:39.254520 18272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:39.254604 18272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.8
I20240502 13:59:39.257840 18272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.217.8
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.18272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.8
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:39.258518 18272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:39.259366 18272 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:39.266053 18280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.266676 18278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:39.267279 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:39.267968 18272 server_base.cc:1034] running on GCE node
I20240502 13:59:39.268296 18272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:39.268842 18272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:39.270004 18272 hybrid_clock.cc:648] HybridClock initialized: now 1714658379269992 us; error 25 us; skew 500 ppm
I20240502 13:59:39.271636 18272 webserver.cc:460] Webserver started at http://127.16.217.8:39273/ using document root <none> and password file <none>
I20240502 13:59:39.272091 18272 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:39.272150 18272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:39.272338 18272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 13:59:39.273689 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.274165 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.276782 17483 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:39.277194 18272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:39.279098 18286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.279906 18272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:39.279996 18272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:39.280068 18272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:39.297983 18272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:39.298491 18272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:39.298624 18272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:39.299049 18272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:39.299819 18272 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:39.299865 18272 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.299912 18272 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:39.299943 18272 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:39.307299 18272 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.8:36031
I20240502 13:59:39.307322 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.8:36031 every 8 connection(s)
I20240502 13:59:39.308104 18272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 13:59:39.315327 18400 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:39.315486 18400 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:39.315878 18400 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:39.316691 17294 ts_manager.cc:194] Registered new tserver with Master: c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031)
I20240502 13:59:39.317214 17294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.8:34297
I20240502 13:59:39.318328 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 18272
I20240502 13:59:39.318429 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 13:59:39.320592 17252 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 13:59:39.330099 17252 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 13:59:39.330181 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17879
I20240502 13:59:39.334972 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 18010
I20240502 13:59:39.341990 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 18141
I20240502 13:59:39.347254 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 18272
I20240502 13:59:39.416303 17614 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:39.559407 17745 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:39.708348 17876 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:43.105422 17294 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:43.123178 17810 tablet_service.cc:1467] Processing CreateTablet for tablet 5b05abb189494b6592787e2f17b4c33a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.123176 17548 tablet_service.cc:1467] Processing CreateTablet for tablet 5b05abb189494b6592787e2f17b4c33a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.123713 17809 tablet_service.cc:1467] Processing CreateTablet for tablet f8f878ad1d9240b79a9b8dbfc50f62d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.124009 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.124011 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.124315 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f878ad1d9240b79a9b8dbfc50f62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.125648 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 30459a30f0de47ba86791dd018538bce (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.126344 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30459a30f0de47ba86791dd018538bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.127815 17678 tablet_service.cc:1467] Processing CreateTablet for tablet f8f878ad1d9240b79a9b8dbfc50f62d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.128060 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f878ad1d9240b79a9b8dbfc50f62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.123171 17547 tablet_service.cc:1467] Processing CreateTablet for tablet 30459a30f0de47ba86791dd018538bce (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.128756 17547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30459a30f0de47ba86791dd018538bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.130287 18420 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:43.130460 17417 tablet_service.cc:1467] Processing CreateTablet for tablet 5b05abb189494b6592787e2f17b4c33a (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.131062 17417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.132052 18420 tablet_bootstrap.cc:654] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.132115 17415 tablet_service.cc:1467] Processing CreateTablet for tablet f8f878ad1d9240b79a9b8dbfc50f62d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.132336 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8f878ad1d9240b79a9b8dbfc50f62d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.132580 17416 tablet_service.cc:1467] Processing CreateTablet for tablet 30459a30f0de47ba86791dd018538bce (DEFAULT_TABLE table=rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.132731 18420 log.cc:824] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:43.132787 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30459a30f0de47ba86791dd018538bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.134130 18420 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:43.134279 18420 ts_tablet_manager.cc:1367] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 13:59:43.135906 18423 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:43.137398 18424 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:43.137652 18420 raft_consensus.cc:348] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.137809 18421 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:43.137837 18420 raft_consensus.cc:374] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.137892 18420 raft_consensus.cc:729] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.138064 18423 tablet_bootstrap.cc:654] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.138317 18420 consensus_queue.cc:260] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.138904 18423 log.cc:824] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:43.139040 18420 ts_tablet_manager.cc:1398] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 13:59:43.139174 18424 tablet_bootstrap.cc:654] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.139775 18424 log.cc:824] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:43.139961 18421 tablet_bootstrap.cc:654] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.140544 18423 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:43.140607 18421 log.cc:824] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:43.140686 18423 ts_tablet_manager.cc:1367] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:43.141105 18424 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:43.141256 18424 ts_tablet_manager.cc:1367] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 13:59:43.141899 18420 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:43.142064 18421 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:43.142186 18421 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 13:59:43.142666 18420 tablet_bootstrap.cc:654] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.143623 18420 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:43.143558 18423 raft_consensus.cc:348] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.143682 18420 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.143718 18423 raft_consensus.cc:374] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.143767 18423 raft_consensus.cc:729] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.144095 18420 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.144193 18420 raft_consensus.cc:374] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.144220 18420 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.144209 18423 consensus_queue.cc:260] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.144340 18420 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.144443 18424 raft_consensus.cc:348] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.144585 18424 raft_consensus.cc:374] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.144615 18420 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.144630 18424 raft_consensus.cc:729] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.144910 18423 ts_tablet_manager.cc:1398] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 13:59:43.145133 18424 consensus_queue.cc:260] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.145260 18421 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.145464 18421 raft_consensus.cc:374] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.145539 18421 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.145875 18424 ts_tablet_manager.cc:1398] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 13:59:43.146191 18421 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.146962 18421 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 13:59:43.147145 18424 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:43.147219 18423 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:43.148088 18424 tablet_bootstrap.cc:654] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.148202 18423 tablet_bootstrap.cc:654] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.149338 18421 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:43.149662 18424 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:43.149731 18424 ts_tablet_manager.cc:1367] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:43.150094 18421 tablet_bootstrap.cc:654] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.150161 18424 raft_consensus.cc:348] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.150276 18424 raft_consensus.cc:374] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.150315 18424 raft_consensus.cc:729] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.150435 18424 consensus_queue.cc:260] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.150687 18424 ts_tablet_manager.cc:1398] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.151059 18421 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:43.151130 18421 ts_tablet_manager.cc:1367] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.151443 18423 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:43.151504 18423 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 13:59:43.151548 18421 raft_consensus.cc:348] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.151635 18421 raft_consensus.cc:374] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.151687 18421 raft_consensus.cc:729] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.151839 18421 consensus_queue.cc:260] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.151984 18423 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.152057 18421 ts_tablet_manager.cc:1398] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.152068 18423 raft_consensus.cc:374] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.152097 18423 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.152215 18423 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.152484 18423 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:43.152612 18423 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:43.153198 18433 raft_consensus.cc:482] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.153393 18423 tablet_bootstrap.cc:654] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.153365 18433 raft_consensus.cc:504] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.154628 18423 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:43.154701 18423 ts_tablet_manager.cc:1367] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.155002 18433 leader_election.cc:288] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.155071 18423 raft_consensus.cc:348] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.155140 18423 raft_consensus.cc:374] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.155164 18423 raft_consensus.cc:729] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.155386 18423 consensus_queue.cc:260] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.155658 18423 ts_tablet_manager.cc:1398] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.158181 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30459a30f0de47ba86791dd018538bce" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:43.158385 17699 raft_consensus.cc:2454] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:43.158515 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30459a30f0de47ba86791dd018538bce" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:43.158738 17437 raft_consensus.cc:2454] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:43.158784 17501 leader_election.cc:302] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:43.158977 18433 raft_consensus.cc:2781] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.159034 18433 raft_consensus.cc:482] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.159076 18433 raft_consensus.cc:3037] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
W20240502 13:59:43.159430 17615 tablet.cc:2369] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:43.159721 18433 raft_consensus.cc:504] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.159981 18433 leader_election.cc:288] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.160224 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30459a30f0de47ba86791dd018538bce" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:43.160249 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30459a30f0de47ba86791dd018538bce" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:43.160339 17437 raft_consensus.cc:3037] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.160339 17699 raft_consensus.cc:3037] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.161237 17699 raft_consensus.cc:2454] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:43.161486 17501 leader_election.cc:302] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:43.161585 17437 raft_consensus.cc:2454] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:43.161643 18433 raft_consensus.cc:2781] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.161828 18433 raft_consensus.cc:686] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Becoming Leader. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Running, Role: LEADER
I20240502 13:59:43.162066 18433 consensus_queue.cc:237] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.164108 17293 catalog_manager.cc:5484] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2). New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:43.200649 17877 tablet.cc:2369] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:43.212956 18431 raft_consensus.cc:482] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.213073 18431 raft_consensus.cc:504] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.213949 18431 leader_election.cc:288] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.217113 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f878ad1d9240b79a9b8dbfc50f62d2" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:43.217162 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f878ad1d9240b79a9b8dbfc50f62d2" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:43.217242 17437 raft_consensus.cc:2454] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:43.217236 17699 raft_consensus.cc:2454] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:43.217540 17763 leader_election.cc:302] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.217728 18431 raft_consensus.cc:2781] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.217789 18431 raft_consensus.cc:482] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.217816 18431 raft_consensus.cc:3037] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.218616 18431 raft_consensus.cc:504] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.218901 18431 leader_election.cc:288] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.219177 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f878ad1d9240b79a9b8dbfc50f62d2" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:43.219187 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8f878ad1d9240b79a9b8dbfc50f62d2" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:43.219261 17437 raft_consensus.cc:3037] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.219274 17699 raft_consensus.cc:3037] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.219861 17437 raft_consensus.cc:2454] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.219897 17699 raft_consensus.cc:2454] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.220400 17764 leader_election.cc:302] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.220548 18431 raft_consensus.cc:2781] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.220614 18431 raft_consensus.cc:686] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:43.220837 18431 consensus_queue.cc:237] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.222999 17293 catalog_manager.cc:5484] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:43.268761 17484 tablet.cc:2369] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 13:59:43.302156 17746 tablet.cc:2369] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 13:59:43.322340 18431 raft_consensus.cc:482] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.322427 18431 raft_consensus.cc:504] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.322759 18431 leader_election.cc:288] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.323298 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:43.323400 17437 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:43.323621 17764 leader_election.cc:302] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.323779 18431 raft_consensus.cc:2781] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.323829 18431 raft_consensus.cc:482] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.323853 18431 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.324590 18431 raft_consensus.cc:504] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.324880 18431 leader_election.cc:288] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:43.325166 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:43.325258 17437 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.325801 17437 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.325958 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:43.326056 17764 leader_election.cc:302] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.326114 17567 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.326133 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:43.326220 18431 raft_consensus.cc:2781] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.326295 18431 raft_consensus.cc:686] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:43.326400 18431 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:43.327013 17567 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.327908 17293 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:43.443898 18446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:43.444062 18446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:43.446537 18446 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 13:59:43.530826 17699 raft_consensus.cc:1261] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.531153 17437 raft_consensus.cc:1261] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.531432 17567 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.531577 17436 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.531698 18431 consensus_queue.cc:1035] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.532032 18431 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.532395 18478 consensus_queue.cc:1035] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.532621 18431 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.534157 17435 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.534551 18433 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.536043 17698 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:43.536417 18433 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:43.540253 18473 mvcc.cc:205] Tried to move back new op lower bound from 7023240738935930880 to 7023240737674395648. Current Snapshot: MvccSnapshot[applied={T|T < 7023240738935930880}]
I20240502 13:59:43.544752 18476 mvcc.cc:205] Tried to move back new op lower bound from 7023240738949554176 to 7023240737433755648. Current Snapshot: MvccSnapshot[applied={T|T < 7023240738949554176}]
I20240502 13:59:43.548662 18483 mvcc.cc:205] Tried to move back new op lower bound from 7023240738935930880 to 7023240737674395648. Current Snapshot: MvccSnapshot[applied={T|T < 7023240738935930880}]
I20240502 13:59:43.577502 18488 mvcc.cc:205] Tried to move back new op lower bound from 7023240738962386944 to 7023240737433755648. Current Snapshot: MvccSnapshot[applied={T|T < 7023240738949554176}]
INSERT report
rows total: 2000
time total: 103.75 ms
time per row: 0.0518751 ms
I20240502 13:59:43.593870 17286 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:43.598438 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 224b4919644d45dc942fb6654eb54c84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.598714 17678 tablet_service.cc:1467] Processing CreateTablet for tablet bb05a5ed865346528523ae99a8d3882b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.598733 17548 tablet_service.cc:1467] Processing CreateTablet for tablet bb05a5ed865346528523ae99a8d3882b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.598985 17547 tablet_service.cc:1467] Processing CreateTablet for tablet 224b4919644d45dc942fb6654eb54c84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.599522 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 4a240577417f4ae1812bac08ba71861b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.599668 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 4a240577417f4ae1812bac08ba71861b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.599773 17810 tablet_service.cc:1467] Processing CreateTablet for tablet 224b4919644d45dc942fb6654eb54c84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.599933 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a240577417f4ae1812bac08ba71861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.599992 17809 tablet_service.cc:1467] Processing CreateTablet for tablet bb05a5ed865346528523ae99a8d3882b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.600114 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 4a240577417f4ae1812bac08ba71861b (DEFAULT_TABLE table=rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:43.598978 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb05a5ed865346528523ae99a8d3882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.602245 18421 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:43.602321 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a240577417f4ae1812bac08ba71861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.602375 18423 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:43.603106 18421 tablet_bootstrap.cc:654] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.603240 18423 tablet_bootstrap.cc:654] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.603870 17547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b4919644d45dc942fb6654eb54c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.600033 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b4919644d45dc942fb6654eb54c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.604411 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb05a5ed865346528523ae99a8d3882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.598731 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b4919644d45dc942fb6654eb54c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.606187 18423 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:43.606280 18423 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:43.606416 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a240577417f4ae1812bac08ba71861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.606473 18420 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:43.606702 18423 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.606806 18423 raft_consensus.cc:374] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.606837 18423 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.606966 18423 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.607116 18420 tablet_bootstrap.cc:654] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.607240 18423 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:43.607614 18424 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:43.608111 18420 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:43.608186 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb05a5ed865346528523ae99a8d3882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:43.608196 18420 ts_tablet_manager.cc:1367] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.608299 18424 tablet_bootstrap.cc:654] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.608628 18420 raft_consensus.cc:348] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.608718 18420 raft_consensus.cc:374] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.608749 18420 raft_consensus.cc:729] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.608861 18420 consensus_queue.cc:260] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.609138 18420 ts_tablet_manager.cc:1398] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.609361 18420 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:43.610201 18420 tablet_bootstrap.cc:654] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.610780 18421 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:43.610849 18421 ts_tablet_manager.cc:1367] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 13:59:43.611150 18420 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:43.611228 18420 ts_tablet_manager.cc:1367] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.611439 18421 raft_consensus.cc:348] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.611533 18421 raft_consensus.cc:374] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.611562 18421 raft_consensus.cc:729] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.611601 18424 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:43.611582 18420 raft_consensus.cc:348] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.611675 18424 ts_tablet_manager.cc:1367] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:43.611685 18420 raft_consensus.cc:374] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.611666 18421 consensus_queue.cc:260] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.611721 18420 raft_consensus.cc:729] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.611851 18420 consensus_queue.cc:260] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.611893 18421 ts_tablet_manager.cc:1398] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:43.612047 18421 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:43.612068 18424 raft_consensus.cc:348] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.612123 18420 ts_tablet_manager.cc:1398] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.612146 18424 raft_consensus.cc:374] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.612176 18424 raft_consensus.cc:729] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.612279 18424 consensus_queue.cc:260] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.612524 18424 ts_tablet_manager.cc:1398] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.612658 18424 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:43.613415 18424 tablet_bootstrap.cc:654] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.613763 18421 tablet_bootstrap.cc:654] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.615818 18424 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:43.615885 18424 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:43.616250 18424 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.616386 18424 raft_consensus.cc:374] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.616418 18424 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.616516 18424 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.616792 18424 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:43.616930 18424 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:43.617605 18424 tablet_bootstrap.cc:654] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.619131 18421 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:43.619215 18421 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 13:59:43.619752 18424 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:43.619830 18424 ts_tablet_manager.cc:1367] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 13:59:43.620219 18424 raft_consensus.cc:348] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.620694 18421 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.620808 18421 raft_consensus.cc:374] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.620854 18421 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.620955 18421 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.621189 18421 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 13:59:43.621250 18424 raft_consensus.cc:374] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.621299 18424 raft_consensus.cc:729] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.621392 18421 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:43.621398 18424 consensus_queue.cc:260] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.621629 18424 ts_tablet_manager.cc:1398] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:43.623003 18421 tablet_bootstrap.cc:654] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:43.623994 18421 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:43.624056 18421 ts_tablet_manager.cc:1367] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 13:59:43.624388 18421 raft_consensus.cc:348] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.624452 18421 raft_consensus.cc:374] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:43.624469 18421 raft_consensus.cc:729] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:43.624575 18421 consensus_queue.cc:260] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.624783 18421 ts_tablet_manager.cc:1398] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:43.713788 18477 raft_consensus.cc:482] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.713881 18477 raft_consensus.cc:504] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.714243 18477 leader_election.cc:288] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:43.714598 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb05a5ed865346528523ae99a8d3882b" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:43.714607 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb05a5ed865346528523ae99a8d3882b" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:43.714715 17699 raft_consensus.cc:2454] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:43.714733 17567 raft_consensus.cc:2454] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:43.715003 17763 leader_election.cc:302] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.715162 18477 raft_consensus.cc:2781] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.715221 18477 raft_consensus.cc:482] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.715269 18477 raft_consensus.cc:3037] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.716033 18477 raft_consensus.cc:504] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.716400 18477 leader_election.cc:288] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:43.716655 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb05a5ed865346528523ae99a8d3882b" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:43.716735 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb05a5ed865346528523ae99a8d3882b" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:43.716776 17699 raft_consensus.cc:3037] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.716836 17567 raft_consensus.cc:3037] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.717705 17567 raft_consensus.cc:2454] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.717705 17699 raft_consensus.cc:2454] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:43.717978 17763 leader_election.cc:302] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:43.718115 18477 raft_consensus.cc:2781] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.718207 18477 raft_consensus.cc:686] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:43.718343 18477 consensus_queue.cc:237] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.720116 17291 catalog_manager.cc:5484] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:43.736402 18432 raft_consensus.cc:482] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.736482 18432 raft_consensus.cc:504] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.737311 18432 leader_election.cc:288] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:43.740474 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a240577417f4ae1812bac08ba71861b" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:43.740609 17437 raft_consensus.cc:2454] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 0.
I20240502 13:59:43.740681 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a240577417f4ae1812bac08ba71861b" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:43.740803 17567 raft_consensus.cc:2454] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 0.
I20240502 13:59:43.740947 17633 leader_election.cc:302] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, b36b507eb389473b8438e907237a63f9; no voters:
I20240502 13:59:43.741154 18432 raft_consensus.cc:2781] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.741202 18432 raft_consensus.cc:482] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.741225 18432 raft_consensus.cc:3037] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.741860 18432 raft_consensus.cc:504] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.742146 18432 leader_election.cc:288] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Requested vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:43.742528 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a240577417f4ae1812bac08ba71861b" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:43.742611 17437 raft_consensus.cc:3037] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.742594 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a240577417f4ae1812bac08ba71861b" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:43.742692 17567 raft_consensus.cc:3037] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.743257 17437 raft_consensus.cc:2454] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 1.
I20240502 13:59:43.743330 17567 raft_consensus.cc:2454] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 1.
I20240502 13:59:43.743518 17633 leader_election.cc:302] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, b36b507eb389473b8438e907237a63f9; no voters:
I20240502 13:59:43.743656 18432 raft_consensus.cc:2781] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.743793 18432 raft_consensus.cc:686] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Becoming Leader. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Running, Role: LEADER
I20240502 13:59:43.744040 18432 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:43.745716 17291 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:43.938519 18440 raft_consensus.cc:482] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:43.938655 18440 raft_consensus.cc:504] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.939100 18440 leader_election.cc:288] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:43.939494 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b4919644d45dc942fb6654eb54c84" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:43.939646 17699 raft_consensus.cc:2454] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:43.939941 17501 leader_election.cc:302] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:43.940094 18440 raft_consensus.cc:2781] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:43.940145 18440 raft_consensus.cc:482] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:43.940171 18440 raft_consensus.cc:3037] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.940974 18440 raft_consensus.cc:504] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.941285 18440 leader_election.cc:288] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:43.941557 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b4919644d45dc942fb6654eb54c84" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:43.941681 17699 raft_consensus.cc:3037] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:43.942567 17699 raft_consensus.cc:2454] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:43.942538 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b4919644d45dc942fb6654eb54c84" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 13:59:43.942538 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224b4919644d45dc942fb6654eb54c84" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85" is_pre_election: true
I20240502 13:59:43.942772 17830 raft_consensus.cc:2454] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:43.942853 17501 leader_election.cc:302] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:43.943014 18440 raft_consensus.cc:2781] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:43.943102 18440 raft_consensus.cc:686] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Becoming Leader. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Running, Role: LEADER
I20240502 13:59:43.943249 18440 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:43.944727 17291 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2). New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:44.053772 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:44.053936 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:44.056396 18522 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 13:59:44.122162 17567 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.122324 17699 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.122514 18477 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.122910 18477 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.124851 17830 raft_consensus.cc:3037] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.125885 17830 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.126889 18433 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.127319 17698 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.127766 18433 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.140321 17568 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.140656 17437 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.141383 18518 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.141721 18518 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 94.8438 ms
time per row: 0.0474219 ms
I20240502 13:59:44.184963 17295 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:44.189469 17547 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38fe2e1de4c0284982425d25829a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.189568 17809 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38fe2e1de4c0284982425d25829a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.189735 17547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d38fe2e1de4c0284982425d25829a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.189810 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d38fe2e1de4c0284982425d25829a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.190536 17416 tablet_service.cc:1467] Processing CreateTablet for tablet be290fd1e9464cc9ba4024b6ec760683 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.190536 17417 tablet_service.cc:1467] Processing CreateTablet for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.190795 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be290fd1e9464cc9ba4024b6ec760683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.190891 17417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.191884 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 37d38fe2e1de4c0284982425d25829a7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.191998 17678 tablet_service.cc:1467] Processing CreateTablet for tablet be290fd1e9464cc9ba4024b6ec760683 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.192049 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37d38fe2e1de4c0284982425d25829a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.192203 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be290fd1e9464cc9ba4024b6ec760683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.192914 18584 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.193312 17547 tablet_service.cc:1467] Processing CreateTablet for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.193539 17547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.193739 18584 tablet_bootstrap.cc:654] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.193781 18583 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.194470 18583 tablet_bootstrap.cc:654] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.194696 17810 tablet_service.cc:1467] Processing CreateTablet for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.194801 18584 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.194866 18584 ts_tablet_manager.cc:1367] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.194890 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c78e509dd8e4e5e8884fa539c9c3f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.195276 18584 raft_consensus.cc:348] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.195363 18584 raft_consensus.cc:374] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.195389 18584 raft_consensus.cc:729] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.195396 18583 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.195464 18583 ts_tablet_manager.cc:1367] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:44.195485 18584 consensus_queue.cc:260] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.195744 18584 ts_tablet_manager.cc:1398] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.195789 18585 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.195909 18584 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.195852 18583 raft_consensus.cc:348] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.195938 18583 raft_consensus.cc:374] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.195963 18583 raft_consensus.cc:729] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.196058 18583 consensus_queue.cc:260] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.196261 18583 ts_tablet_manager.cc:1398] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.196419 18583 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.196559 18585 tablet_bootstrap.cc:654] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.196816 18584 tablet_bootstrap.cc:654] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.197175 18583 tablet_bootstrap.cc:654] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.198706 18585 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.198777 18585 ts_tablet_manager.cc:1367] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:44.198859 18584 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.198925 18584 ts_tablet_manager.cc:1367] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:44.199139 18585 raft_consensus.cc:348] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.199230 18585 raft_consensus.cc:374] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.199168 17809 tablet_service.cc:1467] Processing CreateTablet for tablet be290fd1e9464cc9ba4024b6ec760683 (DEFAULT_TABLE table=rebalance_test_table_2 [id=5df9e9d0ab434adca86a13c5aa4a7f48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.199260 18585 raft_consensus.cc:729] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.199354 18585 consensus_queue.cc:260] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.199455 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be290fd1e9464cc9ba4024b6ec760683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.199564 18585 ts_tablet_manager.cc:1398] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.200191 18584 raft_consensus.cc:348] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.200315 18584 raft_consensus.cc:374] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.200347 18584 raft_consensus.cc:729] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.200475 18584 consensus_queue.cc:260] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.201287 18584 ts_tablet_manager.cc:1398] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.201428 18584 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.201840 18582 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.201996 18583 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.202071 18583 ts_tablet_manager.cc:1367] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:44.202209 18584 tablet_bootstrap.cc:654] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.202421 18583 raft_consensus.cc:348] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.202502 18583 raft_consensus.cc:374] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.202528 18583 raft_consensus.cc:729] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.202626 18583 consensus_queue.cc:260] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.202901 18583 ts_tablet_manager.cc:1398] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.203104 18584 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.203179 18584 ts_tablet_manager.cc:1367] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.203261 18582 tablet_bootstrap.cc:654] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.204232 18584 raft_consensus.cc:348] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.204340 18584 raft_consensus.cc:374] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.204357 18582 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.204373 18584 raft_consensus.cc:729] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.204432 18582 ts_tablet_manager.cc:1367] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:44.204489 18584 consensus_queue.cc:260] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.204753 18584 ts_tablet_manager.cc:1398] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.204888 18582 raft_consensus.cc:348] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.204991 18582 raft_consensus.cc:374] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.205065 18582 raft_consensus.cc:729] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.205204 18582 consensus_queue.cc:260] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.205724 18582 ts_tablet_manager.cc:1398] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.205861 18582 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.206588 18582 tablet_bootstrap.cc:654] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.207621 18582 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.207701 18582 ts_tablet_manager.cc:1367] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.208097 18582 raft_consensus.cc:348] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.208180 18582 raft_consensus.cc:374] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.208200 18582 raft_consensus.cc:729] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.208328 18582 consensus_queue.cc:260] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.208542 18582 ts_tablet_manager.cc:1398] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.208693 18582 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.209350 18582 tablet_bootstrap.cc:654] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.210270 18582 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.210340 18582 ts_tablet_manager.cc:1367] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.210928 18582 raft_consensus.cc:348] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.211011 18582 raft_consensus.cc:374] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.211031 18582 raft_consensus.cc:729] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.211148 18582 consensus_queue.cc:260] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.211347 18582 ts_tablet_manager.cc:1398] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.227443 18477 raft_consensus.cc:482] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.227528 18477 raft_consensus.cc:504] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.227834 18477 leader_election.cc:288] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:44.228134 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be290fd1e9464cc9ba4024b6ec760683" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.228214 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be290fd1e9464cc9ba4024b6ec760683" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:44.228264 17437 raft_consensus.cc:2454] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.228343 17698 raft_consensus.cc:2454] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.228546 17764 leader_election.cc:302] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.228703 18477 raft_consensus.cc:2781] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.228765 18477 raft_consensus.cc:482] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.228818 18477 raft_consensus.cc:3037] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.229460 18477 raft_consensus.cc:504] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.229707 18477 leader_election.cc:288] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:44.229956 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be290fd1e9464cc9ba4024b6ec760683" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.229962 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be290fd1e9464cc9ba4024b6ec760683" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:44.230027 17698 raft_consensus.cc:3037] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.230037 17437 raft_consensus.cc:3037] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.230633 17698 raft_consensus.cc:2454] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.230772 17437 raft_consensus.cc:2454] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.230870 17763 leader_election.cc:302] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.231019 18477 raft_consensus.cc:2781] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.231076 18477 raft_consensus.cc:686] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:44.231182 18477 consensus_queue.cc:237] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.232370 17295 catalog_manager.cc:5484] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:44.273124 18477 raft_consensus.cc:482] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.273203 18477 raft_consensus.cc:504] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.273473 18477 leader_election.cc:288] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.273768 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c78e509dd8e4e5e8884fa539c9c3f6f" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.273777 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c78e509dd8e4e5e8884fa539c9c3f6f" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:44.273860 17437 raft_consensus.cc:2454] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.273878 17566 raft_consensus.cc:2454] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.274117 17764 leader_election.cc:302] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.274291 18477 raft_consensus.cc:2781] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.274348 18477 raft_consensus.cc:482] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.274379 18477 raft_consensus.cc:3037] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.275069 18477 raft_consensus.cc:504] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.275349 18477 leader_election.cc:288] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.275641 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c78e509dd8e4e5e8884fa539c9c3f6f" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.275681 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c78e509dd8e4e5e8884fa539c9c3f6f" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:44.275727 17437 raft_consensus.cc:3037] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.275779 17566 raft_consensus.cc:3037] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.276425 17437 raft_consensus.cc:2454] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.276511 17566 raft_consensus.cc:2454] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.276664 17764 leader_election.cc:302] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.276810 18477 raft_consensus.cc:2781] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.276901 18477 raft_consensus.cc:686] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:44.277009 18477 consensus_queue.cc:237] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.278352 17295 catalog_manager.cc:5484] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:44.326357 18440 raft_consensus.cc:482] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.326457 18440 raft_consensus.cc:504] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.326781 18440 leader_election.cc:288] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.327085 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38fe2e1de4c0284982425d25829a7" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85" is_pre_election: true
I20240502 13:59:44.327088 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38fe2e1de4c0284982425d25829a7" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.327178 17437 raft_consensus.cc:2454] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:44.327185 17830 raft_consensus.cc:2454] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:44.327447 17502 leader_election.cc:302] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b36b507eb389473b8438e907237a63f9; no voters:
I20240502 13:59:44.327615 18440 raft_consensus.cc:2781] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.327668 18440 raft_consensus.cc:482] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.327706 18440 raft_consensus.cc:3037] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.328418 18440 raft_consensus.cc:504] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.328691 18440 leader_election.cc:288] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Requested vote from peers e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.328976 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38fe2e1de4c0284982425d25829a7" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 13:59:44.328966 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37d38fe2e1de4c0284982425d25829a7" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.329082 17437 raft_consensus.cc:3037] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.329082 17830 raft_consensus.cc:3037] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.329646 17830 raft_consensus.cc:2454] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:44.329681 17437 raft_consensus.cc:2454] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:44.329903 17503 leader_election.cc:302] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.330087 18440 raft_consensus.cc:2781] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.330147 18440 raft_consensus.cc:686] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Becoming Leader. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Running, Role: LEADER
I20240502 13:59:44.330260 18440 consensus_queue.cc:237] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.331532 17295 catalog_manager.cc:5484] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2). New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:44.436247 18598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:44.436427 18598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:44.439059 18598 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 13:59:44.506847 17698 raft_consensus.cc:1261] T be290fd1e9464cc9ba4024b6ec760683 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.507336 17437 raft_consensus.cc:1261] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.507651 17435 raft_consensus.cc:1261] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.507936 17566 raft_consensus.cc:1261] T 6c78e509dd8e4e5e8884fa539c9c3f6f P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.508141 18425 consensus_queue.cc:1035] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.508334 18477 consensus_queue.cc:1035] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.511370 18425 consensus_queue.cc:1035] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.511622 18425 consensus_queue.cc:1035] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.513244 17830 raft_consensus.cc:1261] T 37d38fe2e1de4c0284982425d25829a7 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.513639 18433 consensus_queue.cc:1035] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.513715 17436 raft_consensus.cc:1261] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.516705 18626 consensus_queue.cc:1035] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.2111 ms
time per row: 0.0441055 ms
I20240502 13:59:44.562973 17291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:44.567833 17678 tablet_service.cc:1467] Processing CreateTablet for tablet d0d96979d0484944b737aa3e7452947d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.568125 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d96979d0484944b737aa3e7452947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.568204 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 126c4156e9da4208bdba9c3e778f4614 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.568434 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c4156e9da4208bdba9c3e778f4614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.568675 17809 tablet_service.cc:1467] Processing CreateTablet for tablet 126c4156e9da4208bdba9c3e778f4614 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.568908 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c4156e9da4208bdba9c3e778f4614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.570555 18582 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.570621 18585 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.570663 17810 tablet_service.cc:1467] Processing CreateTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.570847 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.571415 18582 tablet_bootstrap.cc:654] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.571673 18585 tablet_bootstrap.cc:654] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.572256 17679 tablet_service.cc:1467] Processing CreateTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.572467 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.572577 18585 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.572635 18585 ts_tablet_manager.cc:1367] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:44.572664 18582 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.572729 18582 ts_tablet_manager.cc:1367] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.573001 18585 raft_consensus.cc:348] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.573035 18582 raft_consensus.cc:348] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.573086 18585 raft_consensus.cc:374] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.573104 18582 raft_consensus.cc:374] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.573114 18585 raft_consensus.cc:729] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.573127 18582 raft_consensus.cc:729] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.573223 18582 consensus_queue.cc:260] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.573228 18585 consensus_queue.cc:260] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.573421 18582 ts_tablet_manager.cc:1398] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.573482 18585 ts_tablet_manager.cc:1398] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.573534 18582 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.573607 18585 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.574153 18582 tablet_bootstrap.cc:654] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.574872 17547 tablet_service.cc:1467] Processing CreateTablet for tablet d0d96979d0484944b737aa3e7452947d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.575110 17547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d96979d0484944b737aa3e7452947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.575254 18585 tablet_bootstrap.cc:654] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.575261 17546 tablet_service.cc:1467] Processing CreateTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.575446 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.577381 18585 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.577376 17415 tablet_service.cc:1467] Processing CreateTablet for tablet d0d96979d0484944b737aa3e7452947d (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.577454 18585 ts_tablet_manager.cc:1367] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 13:59:44.577636 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d96979d0484944b737aa3e7452947d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.577760 18585 raft_consensus.cc:348] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.577821 18585 raft_consensus.cc:374] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.577838 18585 raft_consensus.cc:729] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.577953 18585 consensus_queue.cc:260] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.578169 18585 ts_tablet_manager.cc:1398] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.578271 18585 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.578927 18585 tablet_bootstrap.cc:654] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.579373 18584 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.580039 18584 tablet_bootstrap.cc:654] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.580257 18583 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.580574 18585 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.580749 18585 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.581017 18583 tablet_bootstrap.cc:654] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.581190 18585 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.581290 18585 raft_consensus.cc:374] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.581318 18585 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.581445 18585 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.581733 17416 tablet_service.cc:1467] Processing CreateTablet for tablet 126c4156e9da4208bdba9c3e778f4614 (DEFAULT_TABLE table=rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.581817 18583 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.581872 18583 ts_tablet_manager.cc:1367] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.581920 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126c4156e9da4208bdba9c3e778f4614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.582175 18583 raft_consensus.cc:348] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.582244 18583 raft_consensus.cc:374] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.582266 18583 raft_consensus.cc:729] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.582350 18583 consensus_queue.cc:260] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.582554 18583 ts_tablet_manager.cc:1398] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.582687 18583 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.583088 18585 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.583380 18583 tablet_bootstrap.cc:654] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.584388 18582 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.584388 18584 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.584461 18584 ts_tablet_manager.cc:1367] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 13:59:44.584461 18582 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 13:59:44.584789 18582 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.584789 18584 raft_consensus.cc:348] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.584878 18584 raft_consensus.cc:374] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.584878 18582 raft_consensus.cc:374] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.584913 18584 raft_consensus.cc:729] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.584913 18582 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.585019 18582 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.585026 18584 consensus_queue.cc:260] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.585238 18582 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.585247 18584 ts_tablet_manager.cc:1398] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.585449 18584 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.586100 18584 tablet_bootstrap.cc:654] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.586585 18583 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.586653 18583 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 13:59:44.586903 18584 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.586962 18584 ts_tablet_manager.cc:1367] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.586992 18583 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.587069 18583 raft_consensus.cc:374] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.587093 18583 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.587178 18583 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.587311 18584 raft_consensus.cc:348] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.587383 18584 raft_consensus.cc:374] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.587409 18584 raft_consensus.cc:729] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.587476 18583 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.587519 18584 consensus_queue.cc:260] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.587778 18584 ts_tablet_manager.cc:1398] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.606619 18625 raft_consensus.cc:482] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.606698 18625 raft_consensus.cc:504] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.607018 18625 leader_election.cc:288] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:44.607262 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c4156e9da4208bdba9c3e778f4614" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.607316 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c4156e9da4208bdba9c3e778f4614" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:44.607376 17435 raft_consensus.cc:2454] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.607410 17698 raft_consensus.cc:2454] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:44.607609 17764 leader_election.cc:302] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.607805 18625 raft_consensus.cc:2781] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.607848 18625 raft_consensus.cc:482] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.607916 18625 raft_consensus.cc:3037] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.608007 18626 raft_consensus.cc:482] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.608081 18626 raft_consensus.cc:504] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.608116 18518 raft_consensus.cc:482] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.608178 18518 raft_consensus.cc:504] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.608438 18626 leader_election.cc:288] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.608449 18518 leader_election.cc:288] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.608762 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.608755 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:44.608788 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:44.608850 17435 raft_consensus.cc:2454] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:44.608852 17566 raft_consensus.cc:2454] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 0.
I20240502 13:59:44.608847 18625 raft_consensus.cc:504] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.609072 17502 leader_election.cc:302] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b36b507eb389473b8438e907237a63f9; no voters:
I20240502 13:59:44.609071 17633 leader_election.cc:302] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:44.609171 18625 leader_election.cc:288] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:44.609285 18626 raft_consensus.cc:2781] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.609333 18626 raft_consensus.cc:482] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.609373 18626 raft_consensus.cc:3037] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.609378 18518 raft_consensus.cc:2781] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.609441 18518 raft_consensus.cc:482] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.609488 18518 raft_consensus.cc:3037] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.609470 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c4156e9da4208bdba9c3e778f4614" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.609549 17435 raft_consensus.cc:3037] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.610059 18626 raft_consensus.cc:504] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.610261 17435 raft_consensus.cc:2454] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.610261 18518 raft_consensus.cc:504] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.610327 18626 leader_election.cc:288] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.610502 17764 leader_election.cc:302] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.610555 18518 leader_election.cc:288] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:44.610648 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.610720 17435 raft_consensus.cc:3037] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.610852 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:44.611018 17566 raft_consensus.cc:2379] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b0da919a0055405e8ef4dfd545e3bd23 in current term 1: Already voted for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in this term.
I20240502 13:59:44.611029 17437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:44.611274 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:44.611416 17699 raft_consensus.cc:2379] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c2e9843dbba4851ad72d2d4a8b9886b in current term 1: Already voted for candidate b0da919a0055405e8ef4dfd545e3bd23 in this term.
I20240502 13:59:44.611500 17435 raft_consensus.cc:2454] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:44.611632 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d96979d0484944b737aa3e7452947d" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:44.611685 18625 raft_consensus.cc:2781] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.611745 17697 raft_consensus.cc:2379] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7c2e9843dbba4851ad72d2d4a8b9886b in current term 1: Already voted for candidate b0da919a0055405e8ef4dfd545e3bd23 in this term.
I20240502 13:59:44.611769 18625 raft_consensus.cc:686] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:44.611876 17502 leader_election.cc:302] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b36b507eb389473b8438e907237a63f9; no voters:
I20240502 13:59:44.611888 18625 consensus_queue.cc:237] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.611974 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126c4156e9da4208bdba9c3e778f4614" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:44.612028 18626 raft_consensus.cc:2781] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.612051 17698 raft_consensus.cc:3037] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.612094 18626 raft_consensus.cc:686] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Becoming Leader. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Running, Role: LEADER
I20240502 13:59:44.612187 18626 consensus_queue.cc:237] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 13:59:44.612779 17698 raft_consensus.cc:2454] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:44.613322 17633 leader_election.cc:302] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23; no voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b36b507eb389473b8438e907237a63f9
I20240502 13:59:44.613457 18518 raft_consensus.cc:2726] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 13:59:44.613390 17293 catalog_manager.cc:5484] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:44.613665 17291 catalog_manager.cc:5484] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2). New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:44.627393 18518 raft_consensus.cc:482] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.627466 18518 raft_consensus.cc:504] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.627801 18518 leader_election.cc:288] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:44.628054 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:44.628155 17566 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 0.
I20240502 13:59:44.628427 17633 leader_election.cc:302] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:44.628563 18518 raft_consensus.cc:2781] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.628609 18518 raft_consensus.cc:482] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.628649 18518 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.629351 18518 raft_consensus.cc:504] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.629621 18518 leader_election.cc:288] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:44.629844 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:44.629943 17566 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.630647 17566 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 1.
I20240502 13:59:44.630873 17633 leader_election.cc:302] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:44.631008 18518 raft_consensus.cc:2781] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.631114 18518 raft_consensus.cc:686] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Becoming Leader. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Running, Role: LEADER
I20240502 13:59:44.631306 17830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85" is_pre_election: true
I20240502 13:59:44.631367 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 13:59:44.631390 17830 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0da919a0055405e8ef4dfd545e3bd23 in term 0.
I20240502 13:59:44.631424 18518 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.632596 17291 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:44.734120 18651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:44.734297 18651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:44.736835 18651 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 13:59:44.799041 17698 raft_consensus.cc:1261] T 126c4156e9da4208bdba9c3e778f4614 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.799386 17435 raft_consensus.cc:1261] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.799916 18477 consensus_queue.cc:1035] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.800611 18477 consensus_queue.cc:1035] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.801522 17437 raft_consensus.cc:1261] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.801560 17697 raft_consensus.cc:1261] T d0d96979d0484944b737aa3e7452947d P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.801863 18433 consensus_queue.cc:1035] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.805871 18433 consensus_queue.cc:1035] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.818791 17829 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.819818 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.820101 17566 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:44.820748 18432 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:44.821043 18432 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.9377 ms
time per row: 0.0444688 ms
I20240502 13:59:44.861389 17291 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40584:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 13:59:44.865971 17416 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7f702fa6854711b6afbebd76662177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.865979 17546 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7f702fa6854711b6afbebd76662177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.866086 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7f702fa6854711b6afbebd76662177 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.866216 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7f702fa6854711b6afbebd76662177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.866216 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7f702fa6854711b6afbebd76662177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.866317 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7f702fa6854711b6afbebd76662177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.867739 18585 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.868119 18584 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.868345 18583 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.868350 17679 tablet_service.cc:1467] Processing CreateTablet for tablet 588123f560c04bc4b9a3d3c5c4251135 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.868579 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588123f560c04bc4b9a3d3c5c4251135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.868873 18585 tablet_bootstrap.cc:654] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.868955 18584 tablet_bootstrap.cc:654] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.869191 18583 tablet_bootstrap.cc:654] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.869954 17810 tablet_service.cc:1467] Processing CreateTablet for tablet a67fbce1caca46fa818f4e5d90a15305 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.870021 17809 tablet_service.cc:1467] Processing CreateTablet for tablet 588123f560c04bc4b9a3d3c5c4251135 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.870085 18584 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.870155 18584 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.870193 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.870570 18584 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.870662 18584 raft_consensus.cc:374] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.870692 18584 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.870788 18584 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.871021 18584 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.871104 17415 tablet_service.cc:1467] Processing CreateTablet for tablet 588123f560c04bc4b9a3d3c5c4251135 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.871301 17415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588123f560c04bc4b9a3d3c5c4251135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.872059 18585 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.872120 18585 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 13:59:44.872411 18582 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.872519 18585 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.872594 18585 raft_consensus.cc:374] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.872622 18585 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.872723 18585 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.872944 18585 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.873009 18584 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.873145 18585 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23: Bootstrap starting.
I20240502 13:59:44.873143 18582 tablet_bootstrap.cc:654] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.873689 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588123f560c04bc4b9a3d3c5c4251135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.873759 18584 tablet_bootstrap.cc:654] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.873884 18585 tablet_bootstrap.cc:654] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.874758 18584 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.874835 18584 ts_tablet_manager.cc:1367] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:44.875061 18582 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.875141 18582 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 13:59:44.875226 18584 raft_consensus.cc:348] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.875312 18584 raft_consensus.cc:374] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.875346 18584 raft_consensus.cc:729] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.875437 18584 consensus_queue.cc:260] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.875524 18582 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.875607 18582 raft_consensus.cc:374] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.875633 18582 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.875658 18584 ts_tablet_manager.cc:1398] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.875720 18582 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.875944 18582 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.876102 18582 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85: Bootstrap starting.
I20240502 13:59:44.876686 18583 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.876761 18583 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 13:59:44.876909 18582 tablet_bootstrap.cc:654] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.877069 18583 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.877135 18583 raft_consensus.cc:374] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.877152 18583 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.877271 18583 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:44.877501 18583 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.877835 17546 tablet_service.cc:1467] Processing CreateTablet for tablet a67fbce1caca46fa818f4e5d90a15305 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.878038 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.879369 18585 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23: No bootstrap required, opened a new log
I20240502 13:59:44.879437 18585 ts_tablet_manager.cc:1367] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 13:59:44.879827 18585 raft_consensus.cc:348] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.879909 18585 raft_consensus.cc:374] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.879936 18585 raft_consensus.cc:729] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.880053 18585 consensus_queue.cc:260] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.880357 18585 ts_tablet_manager.cc:1398] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.880662 18582 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85: No bootstrap required, opened a new log
I20240502 13:59:44.880733 18582 ts_tablet_manager.cc:1367] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 13:59:44.881156 18582 raft_consensus.cc:348] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.881156 17416 tablet_service.cc:1467] Processing CreateTablet for tablet a67fbce1caca46fa818f4e5d90a15305 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 13:59:44.881243 18582 raft_consensus.cc:374] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.881278 18582 raft_consensus.cc:729] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.881364 17416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:44.881392 18582 consensus_queue.cc:260] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.881855 18582 ts_tablet_manager.cc:1398] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 13:59:44.882920 18584 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 13:59:44.883111 18583 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Bootstrap starting.
I20240502 13:59:44.883867 18583 tablet_bootstrap.cc:654] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.883867 18584 tablet_bootstrap.cc:654] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Neither blocks nor log segments found. Creating new log.
I20240502 13:59:44.884760 18583 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: No bootstrap required, opened a new log
I20240502 13:59:44.884760 18584 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: No bootstrap required, opened a new log
I20240502 13:59:44.884835 18584 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 13:59:44.884835 18583 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 13:59:44.885167 18583 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.885167 18584 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.885257 18583 raft_consensus.cc:374] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.885257 18584 raft_consensus.cc:374] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 13:59:44.885293 18583 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.885293 18584 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 13:59:44.885396 18583 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.885396 18584 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.885604 18584 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 13:59:44.885614 18583 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 13:59:44.964494 18680 raft_consensus.cc:482] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:44.964588 18680 raft_consensus.cc:504] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.965591 18680 leader_election.cc:288] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:44.968748 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "b36b507eb389473b8438e907237a63f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85" is_pre_election: true
I20240502 13:59:44.968807 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "b36b507eb389473b8438e907237a63f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:44.968865 17829 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36b507eb389473b8438e907237a63f9 in term 0.
I20240502 13:59:44.968935 17698 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36b507eb389473b8438e907237a63f9 in term 0.
I20240502 13:59:44.969239 17372 leader_election.cc:302] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.969461 18680 raft_consensus.cc:2781] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:44.969519 18680 raft_consensus.cc:482] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:44.969561 18680 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.970239 18680 raft_consensus.cc:504] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.970533 18680 leader_election.cc:288] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [CANDIDATE]: Term 1 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:44.970794 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "b36b507eb389473b8438e907237a63f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:44.970827 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "b36b507eb389473b8438e907237a63f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 13:59:44.970892 17698 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.970906 17829 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:44.971647 17829 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36b507eb389473b8438e907237a63f9 in term 1.
I20240502 13:59:44.971647 17698 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36b507eb389473b8438e907237a63f9 in term 1.
I20240502 13:59:44.971874 17372 leader_election.cc:302] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:44.972038 18680 raft_consensus.cc:2781] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:44.972102 18680 raft_consensus.cc:686] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 LEADER]: Becoming Leader. State: Replica: b36b507eb389473b8438e907237a63f9, State: Running, Role: LEADER
I20240502 13:59:44.972357 18680 consensus_queue.cc:237] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:44.974195 17291 catalog_manager.cc:5484] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b36b507eb389473b8438e907237a63f9 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "b36b507eb389473b8438e907237a63f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:45.003345 18626 raft_consensus.cc:482] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.003424 18626 raft_consensus.cc:504] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:45.003746 18626 leader_election.cc:288] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:45.004019 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7f702fa6854711b6afbebd76662177" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:45.004026 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7f702fa6854711b6afbebd76662177" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:45.004117 17435 raft_consensus.cc:2454] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:45.004127 17698 raft_consensus.cc:2454] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 0.
I20240502 13:59:45.004359 17501 leader_election.cc:302] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:45.004494 18626 raft_consensus.cc:2781] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.004545 18626 raft_consensus.cc:482] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.004590 18626 raft_consensus.cc:3037] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.005278 18626 raft_consensus.cc:504] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:45.005558 18626 leader_election.cc:288] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Requested vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 13:59:45.005808 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7f702fa6854711b6afbebd76662177" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:45.005822 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7f702fa6854711b6afbebd76662177" candidate_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:45.005889 17698 raft_consensus.cc:3037] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.005896 17435 raft_consensus.cc:3037] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.006484 17698 raft_consensus.cc:2454] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:45.006619 17435 raft_consensus.cc:2454] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2e9843dbba4851ad72d2d4a8b9886b in term 1.
I20240502 13:59:45.006740 17501 leader_election.cc:302] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23; no voters:
I20240502 13:59:45.006882 18626 raft_consensus.cc:2781] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.006961 18626 raft_consensus.cc:686] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Becoming Leader. State: Replica: 7c2e9843dbba4851ad72d2d4a8b9886b, State: Running, Role: LEADER
I20240502 13:59:45.007074 18626 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:45.008540 17291 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2). New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:45.085167 18477 raft_consensus.cc:482] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 13:59:45.085268 18477 raft_consensus.cc:504] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:45.085569 18477 leader_election.cc:288] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:45.085877 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" is_pre_election: true
I20240502 13:59:45.085944 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9" is_pre_election: true
I20240502 13:59:45.086019 17566 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:45.086050 17435 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 0.
I20240502 13:59:45.086338 17764 leader_election.cc:302] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:45.086484 18477 raft_consensus.cc:2781] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 13:59:45.086544 18477 raft_consensus.cc:482] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 13:59:45.086591 18477 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.087335 18477 raft_consensus.cc:504] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:45.087643 18477 leader_election.cc:288] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:45.087955 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 13:59:45.088016 17435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36b507eb389473b8438e907237a63f9"
I20240502 13:59:45.088096 17566 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.088112 17435 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 13:59:45.089018 17566 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:45.089018 17435 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:45.089318 17764 leader_election.cc:302] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 13:59:45.089462 18477 raft_consensus.cc:2781] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 13:59:45.089567 18477 raft_consensus.cc:686] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:45.089707 18477 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:45.091240 17291 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
W20240502 13:59:45.195071 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.195245 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.197783 18706 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 13:59:45.267717 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.267817 17435 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.268388 18440 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.272425 17437 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.273240 18625 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.273783 17566 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.276839 17829 raft_consensus.cc:1261] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer b36b507eb389473b8438e907237a63f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.277210 18680 consensus_queue.cc:1035] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.281955 17698 raft_consensus.cc:1261] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer b36b507eb389473b8438e907237a63f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 13:59:45.282267 18680 consensus_queue.cc:1035] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.292236 18626 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.293151 18425 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 13:59:45.294497 18633 mvcc.cc:205] Tried to move back new op lower bound from 7023240746051813376 to 7023240744989995008. Current Snapshot: MvccSnapshot[applied={T|T < 7023240746051813376}]
I20240502 13:59:45.303503 18686 mvcc.cc:205] Tried to move back new op lower bound from 7023240746051813376 to 7023240744989995008. Current Snapshot: MvccSnapshot[applied={T|T < 7023240746051813376}]
INSERT report
rows total: 2000
time total: 99.154 ms
time per row: 0.049577 ms
I20240502 13:59:45.337754 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:35713
--local_ip_for_outbound_sockets=127.16.217.5
--tserver_master_addrs=127.16.217.62:46605
--webserver_port=41331
--webserver_interface=127.16.217.5
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:45.435947 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.436133 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:45.436187 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.438913 18743 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:45.438959 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:45.439028 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 13:59:45.441921 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:35713
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=41331
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.18743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:45.442512 18743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:45.443325 18743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:45.450064 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.450471 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.451079 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:45.451700 18743 server_base.cc:1034] running on GCE node
I20240502 13:59:45.451961 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:45.452383 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:45.453513 18743 hybrid_clock.cc:648] HybridClock initialized: now 1714658385453500 us; error 24 us; skew 500 ppm
I20240502 13:59:45.455040 18743 webserver.cc:460] Webserver started at http://127.16.217.5:41331/ using document root <none> and password file <none>
I20240502 13:59:45.455564 18743 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:45.455646 18743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:45.458446 18743 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 13:59:45.459949 18757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.460738 18743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.460820 18743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3305e392697b48a8ad63900628ce4c27"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:45.461112 18743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:45.477630 18743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:45.478087 18743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:45.478349 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:45.479036 18743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:45.479681 18743 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:45.479719 18743 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.479753 18743 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:45.479779 18743 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.486409 18743 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:35713
I20240502 13:59:45.486428 18870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:35713 every 8 connection(s)
I20240502 13:59:45.487141 18743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 13:59:45.492528 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 18743
I20240502 13:59:45.492897 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:40157
--local_ip_for_outbound_sockets=127.16.217.6
--tserver_master_addrs=127.16.217.62:46605
--webserver_port=45669
--webserver_interface=127.16.217.6
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 13:59:45.494166 18871 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:45.494329 18871 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:45.494709 18871 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:45.495649 17295 ts_manager.cc:194] Re-registered known tserver with Master: 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713)
I20240502 13:59:45.496351 17295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:35017
W20240502 13:59:45.583113 18874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.583302 18874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:45.583357 18874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.585878 18874 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:45.585927 18874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:45.585999 18874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 13:59:45.589356 18874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:40157
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=45669
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.18874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:45.589949 18874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:45.590837 18874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:45.597528 18882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.597955 18880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.598572 18879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:45.599128 18874 server_base.cc:1034] running on GCE node
I20240502 13:59:45.599382 18874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:45.599777 18874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:45.600905 18874 hybrid_clock.cc:648] HybridClock initialized: now 1714658385600886 us; error 33 us; skew 500 ppm
I20240502 13:59:45.602294 18874 webserver.cc:460] Webserver started at http://127.16.217.6:45669/ using document root <none> and password file <none>
I20240502 13:59:45.602707 18874 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:45.602769 18874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:45.605352 18874 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.606806 18888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.607648 18874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.607767 18874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "73731152a18648aab3b1a720dabcc26b"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 13:59:45.608096 18874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:45.618456 18874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:45.618932 18874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:45.619064 18874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:45.619493 18874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:45.620182 18874 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:45.620227 18874 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.620296 18874 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:45.620321 18874 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.627482 18874 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:40157
I20240502 13:59:45.627498 19001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:40157 every 8 connection(s)
I20240502 13:59:45.628576 18874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 13:59:45.635705 19002 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:45.635864 19002 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:45.636240 19002 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:45.637060 17295 ts_manager.cc:194] Re-registered known tserver with Master: 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157)
I20240502 13:59:45.637526 17295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:54691
I20240502 13:59:45.638334 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 18874
I20240502 13:59:45.638762 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:44123
--local_ip_for_outbound_sockets=127.16.217.7
--tserver_master_addrs=127.16.217.62:46605
--webserver_port=42705
--webserver_interface=127.16.217.7
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:45.733908 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.734069 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:45.734112 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.736698 19005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:45.736749 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:45.736840 19005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 13:59:45.740154 19005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:44123
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=42705
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.19005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:45.740811 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:45.741693 19005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:45.748731 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.749157 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.749722 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:45.750353 19005 server_base.cc:1034] running on GCE node
I20240502 13:59:45.750634 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:45.751065 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:45.752224 19005 hybrid_clock.cc:648] HybridClock initialized: now 1714658385752207 us; error 27 us; skew 500 ppm
I20240502 13:59:45.753764 19005 webserver.cc:460] Webserver started at http://127.16.217.7:42705/ using document root <none> and password file <none>
I20240502 13:59:45.754225 19005 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:45.754290 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:45.757077 19005 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 13:59:45.758675 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.759541 19005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.759631 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:45.759960 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:45.782980 19005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:45.783460 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:45.783589 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:45.784049 19005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:45.784848 19005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:45.784895 19005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.784945 19005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:45.784972 19005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.793097 19005 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:44123
I20240502 13:59:45.793125 19132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:44123 every 8 connection(s)
I20240502 13:59:45.793938 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 13:59:45.801570 19133 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:45.801766 19133 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:45.802202 19133 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:45.803066 17295 ts_manager.cc:194] Re-registered known tserver with Master: d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123)
I20240502 13:59:45.803489 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 19005
I20240502 13:59:45.803591 17295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:60467
I20240502 13:59:45.803936 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.8:36031
--local_ip_for_outbound_sockets=127.16.217.8
--tserver_master_addrs=127.16.217.62:46605
--webserver_port=39273
--webserver_interface=127.16.217.8
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 13:59:45.910017 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:45.910199 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 13:59:45.910265 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:45.913251 19136 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 13:59:45.913312 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 13:59:45.913436 19136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.8
I20240502 13:59:45.916837 19136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.8:36031
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.217.8
--webserver_port=39273
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.19136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.8
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 13:59:45.917527 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 13:59:45.918494 19136 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 13:59:45.926115 19144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.926636 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 13:59:45.927150 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 13:59:45.927770 19136 server_base.cc:1034] running on GCE node
I20240502 13:59:45.928130 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 13:59:45.928625 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 13:59:45.929785 19136 hybrid_clock.cc:648] HybridClock initialized: now 1714658385929767 us; error 37 us; skew 500 ppm
I20240502 13:59:45.931485 19136 webserver.cc:460] Webserver started at http://127.16.217.8:39273/ using document root <none> and password file <none>
I20240502 13:59:45.931978 19136 fs_manager.cc:359] Metadata directory not provided
I20240502 13:59:45.932060 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 13:59:45.934954 19136 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.936702 19150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.937551 19136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 13:59:45.937652 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
format_stamp: "Formatted at 2024-05-02 13:59:39 on dist-test-slave-7msf"
I20240502 13:59:45.937983 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 13:59:45.953771 19136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 13:59:45.954237 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 13:59:45.954368 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 13:59:45.954797 19136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 13:59:45.955499 19136 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 13:59:45.955544 19136 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.955579 19136 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 13:59:45.955614 19136 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 13:59:45.963304 19136 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.8:36031
I20240502 13:59:45.963367 19263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.8:36031 every 8 connection(s)
I20240502 13:59:45.964340 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 13:59:45.969311 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 19136
I20240502 13:59:45.972208 19264 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 13:59:45.972419 19264 heartbeater.cc:464] Registering TS with master...
I20240502 13:59:45.972944 19264 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 13:59:45.973920 17295 ts_manager.cc:194] Re-registered known tserver with Master: c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031)
I20240502 13:59:45.974673 17295 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.8:55433
I20240502 13:59:46.137612 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.137719 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.138820 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:46.142382 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.143676 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.157717 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.164548 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.165158 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.316546 17295 ts_manager.cc:292] Set tserver state for b36b507eb389473b8438e907237a63f9 to MAINTENANCE_MODE
W20240502 13:59:46.445588 19327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.445783 19327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.448444 19327 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:46.497947 18871 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:46.532073 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.532214 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.533783 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.543972 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:46.550626 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.558270 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.560456 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.561967 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.569770 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.570495 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.578722 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.579356 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.588958 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:46.591262 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.608554 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.613070 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.639477 19002 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:46.715381 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.719909 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.722550 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.725876 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.729645 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.748561 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.752182 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.756675 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 13:59:46.797559 19421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:46.797803 19421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:46.801508 19421 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:46.805634 19133 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:46.849205 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.853155 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.854667 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:46.861308 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.868534 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.869400 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.869825 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.885900 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.916323 17829 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } }
I20240502 13:59:46.917783 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 13:59:46.918177 17437 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 13:59:46.918799 18425 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 13:59:46.919123 18425 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 13:59:46.922034 18425 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } }
I20240502 13:59:46.925418 17288 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:46.925844 17437 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } }
I20240502 13:59:46.930424 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:46.930874 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } }
I20240502 13:59:46.932341 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:46.933769 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:46.934062 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:46.935099 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.947607 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.954727 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.955087 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:46.976433 19264 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 13:59:46.980388 17829 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 13:59:46.981619 17437 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 13:59:46.981966 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 13:59:46.981966 19537 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:46.982290 19537 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:46.983999 19531 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 13:59:46.984670 17437 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 13:59:46.985564 17281 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5b05abb189494b6592787e2f17b4c33a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 13:59:46.986012 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
W20240502 13:59:46.987311 17766 consensus_peers.cc:485] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 -> Peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157): Couldn't send request to peer 73731152a18648aab3b1a720dabcc26b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b05abb189494b6592787e2f17b4c33a. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:46.987547 17291 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:47.108942 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.109154 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.111927 19543 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:47.179735 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.181031 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.181414 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:47.182005 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:47.203426 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.205451 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.210479 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:47.222220 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 13:59:47.360066 19585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.360260 19585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.362938 19585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:47.434084 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.434238 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.437986 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.445647 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:47.447037 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 13:59:47.452761 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.469558 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:47.472344 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:47.599730 19633 ts_tablet_manager.cc:902] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:47.600409 19633 tablet_copy_client.cc:308] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 13:59:47.604605 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet 5b05abb189494b6592787e2f17b4c33a from peer 73731152a18648aab3b1a720dabcc26b ({username='slave'} at 127.16.217.6:39333)
I20240502 13:59:47.604740 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet 5b05abb189494b6592787e2f17b4c33a from peer 73731152a18648aab3b1a720dabcc26b at {username='slave'} at 127.16.217.6:39333: session id = 73731152a18648aab3b1a720dabcc26b-5b05abb189494b6592787e2f17b4c33a
I20240502 13:59:47.605859 17850 tablet_copy_source_session.cc:205] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:47.608757 19633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:47.613883 19633 tablet_copy_client.cc:783] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:47.614096 19633 tablet_copy_client.cc:647] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:47.615813 19637 tablet_copy_client.cc:1002] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: tablet copy: Injecting latency into file download: 1500
W20240502 13:59:47.624789 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 13:59:47.624960 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 13:59:47.627593 19627 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 13:59:47.696650 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.700462 17416 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 13:59:47.702577 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 13:59:47.703923 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.704767 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.708082 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 13:59:47.716671 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 13:59:47.744742 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 13:59:47.777159 19326 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17355
W20240502 13:59:47.801484 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:47.802361 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:47.818900 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:47.819904 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:47.851982 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:47.949656 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:47.987046 17764 consensus_peers.cc:485] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.025158 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.055557 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.058630 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.071481 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.106837 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:48.242457 17633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:48.243180 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:48.947466 19711 raft_consensus.cc:482] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b36b507eb389473b8438e907237a63f9)
I20240502 13:59:48.947623 19711 raft_consensus.cc:504] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:48.948092 19711 leader_election.cc:288] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:48.948561 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "b0da919a0055405e8ef4dfd545e3bd23" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e7f378d8d4c745d888ed99454393bb85" is_pre_election: true
W20240502 13:59:48.949183 17633 leader_election.cc:334] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:48.949281 17633 leader_election.cc:302] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23; no voters: b36b507eb389473b8438e907237a63f9, e7f378d8d4c745d888ed99454393bb85
I20240502 13:59:48.949422 19711 raft_consensus.cc:2726] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 13:59:49.081162 19537 raft_consensus.cc:482] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b36b507eb389473b8438e907237a63f9)
I20240502 13:59:49.081295 19537 raft_consensus.cc:504] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:49.081677 19537 leader_election.cc:288] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:49.082119 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23" is_pre_election: true
I20240502 13:59:49.082283 17697 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 1.
I20240502 13:59:49.082584 17763 leader_election.cc:302] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters:
W20240502 13:59:49.082722 17764 leader_election.cc:334] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:49.082733 19537 raft_consensus.cc:2781] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 13:59:49.082816 19537 raft_consensus.cc:482] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b36b507eb389473b8438e907237a63f9)
I20240502 13:59:49.082861 19537 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 13:59:49.083819 19537 raft_consensus.cc:504] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:49.084110 19537 leader_election.cc:288] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 election: Requested vote from peers b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 13:59:49.084475 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588123f560c04bc4b9a3d3c5c4251135" candidate_uuid: "e7f378d8d4c745d888ed99454393bb85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 13:59:49.084587 17697 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Advancing to term 2
W20240502 13:59:49.084971 17764 leader_election.cc:334] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:49.085424 17697 raft_consensus.cc:2454] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f378d8d4c745d888ed99454393bb85 in term 2.
I20240502 13:59:49.085664 17763 leader_election.cc:302] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b0da919a0055405e8ef4dfd545e3bd23, e7f378d8d4c745d888ed99454393bb85; no voters: b36b507eb389473b8438e907237a63f9
I20240502 13:59:49.085837 19684 raft_consensus.cc:2781] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 13:59:49.085922 19684 raft_consensus.cc:686] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [term 2 LEADER]: Becoming Leader. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 13:59:49.086056 19684 consensus_queue.cc:237] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:49.088215 17288 catalog_manager.cc:5484] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: term changed from 1 to 2, leader changed from b36b507eb389473b8438e907237a63f9 (127.16.217.1) to e7f378d8d4c745d888ed99454393bb85 (127.16.217.4). New cstate: current_term: 2 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:49.116294 19633 tablet_copy_client.cc:523] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:49.118088 19633 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Bootstrap starting.
I20240502 13:59:49.123039 19633 log.cc:824] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:49.140631 19633 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 13:59:49.140937 19633 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Bootstrap complete.
I20240502 13:59:49.141247 19633 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20240502 13:59:49.143545 19633 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 13:59:49.143750 19633 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Initialized, Role: LEARNER
I20240502 13:59:49.144052 19633 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 13:59:49.144829 19633 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 13:59:49.145701 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session 73731152a18648aab3b1a720dabcc26b-5b05abb189494b6592787e2f17b4c33a received from {username='slave'} at 127.16.217.6:39333
I20240502 13:59:49.145803 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session 73731152a18648aab3b1a720dabcc26b-5b05abb189494b6592787e2f17b4c33a on tablet 5b05abb189494b6592787e2f17b4c33a with peer 73731152a18648aab3b1a720dabcc26b
I20240502 13:59:49.505323 18956 raft_consensus.cc:1203] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 13:59:49.573206 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:49.577162 17697 raft_consensus.cc:1261] T 588123f560c04bc4b9a3d3c5c4251135 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 13:59:49.577517 19537 consensus_queue.cc:1035] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 13:59:50.083669 19762 raft_consensus.cc:1050] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER 73731152a18648aab3b1a720dabcc26b to VOTER
I20240502 13:59:50.084122 19762 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 13:59:50.085662 18956 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 13:59:50.085784 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 13:59:50.086047 17764 consensus_peers.cc:485] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:50.086119 19761 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:50.086272 19764 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:50.088729 19761 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.089294 18956 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.103670 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.104401 17291 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.107940 17829 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 13:59:50.108902 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:50.108947 18956 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:50.109329 19764 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:50.109460 19761 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:50.110833 19764 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.111270 18956 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 13:59:50.111634 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
W20240502 13:59:50.124053 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:50.202729 17281 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b05abb189494b6592787e2f17b4c33a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 13:59:50.204254 17291 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
W20240502 13:59:50.207139 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:50.207927 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:50.304261 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.347597 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.387981 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.402127 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.406440 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.461436 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:50.491725 17566 consensus_queue.cc:237] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } }
I20240502 13:59:50.492983 17697 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 13:59:50.493269 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:50.493355 19788 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 13:59:50.494968 19698 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } }
I20240502 13:59:50.495360 17697 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } }
I20240502 13:59:50.510711 17288 catalog_manager.cc:5484] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:50.513918 17566 consensus_queue.cc:237] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 13:59:50.515290 17697 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 13:59:50.515671 19698 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:50.517226 19790 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
W20240502 13:59:50.517778 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:50.517663 17697 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
W20240502 13:59:50.519958 17504 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123): Couldn't send request to peer d2dc7e8c2808471fb5a9a951c70d5c2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30459a30f0de47ba86791dd018538bce. This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:50.594419 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:50.612993 17280 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 30459a30f0de47ba86791dd018538bce with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 13:59:50.614255 17288 catalog_manager.cc:5484] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 4 to 5, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
W20240502 13:59:50.758504 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:50.766880 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:51.115234 19809 ts_tablet_manager.cc:902] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Initiating tablet copy from peer 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:51.116022 19809 tablet_copy_client.cc:308] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:42667
I20240502 13:59:51.119235 17588 tablet_copy_service.cc:140] P 7c2e9843dbba4851ad72d2d4a8b9886b: Received BeginTabletCopySession request for tablet 30459a30f0de47ba86791dd018538bce from peer d2dc7e8c2808471fb5a9a951c70d5c2a ({username='slave'} at 127.16.217.7:41287)
I20240502 13:59:51.119354 17588 tablet_copy_service.cc:161] P 7c2e9843dbba4851ad72d2d4a8b9886b: Beginning new tablet copy session on tablet 30459a30f0de47ba86791dd018538bce from peer d2dc7e8c2808471fb5a9a951c70d5c2a at {username='slave'} at 127.16.217.7:41287: session id = d2dc7e8c2808471fb5a9a951c70d5c2a-30459a30f0de47ba86791dd018538bce
I20240502 13:59:51.120487 17588 tablet_copy_source_session.cc:205] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:51.123641 19809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30459a30f0de47ba86791dd018538bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:51.128918 19809 tablet_copy_client.cc:783] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:51.129104 19809 tablet_copy_client.cc:647] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:51.130527 19813 tablet_copy_client.cc:1002] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Injecting latency into file download: 1500
W20240502 13:59:51.417757 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:52.267679 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:52.486220 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:52.566962 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 13:59:52.630996 19809 tablet_copy_client.cc:523] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:52.720501 19809 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap starting.
I20240502 13:59:52.726665 19809 log.cc:824] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:52.747676 19809 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 13:59:52.748044 19809 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap complete.
I20240502 13:59:52.748409 19809 ts_tablet_manager.cc:1367] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.007s
I20240502 13:59:52.750290 19809 raft_consensus.cc:348] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 13:59:52.750504 19809 raft_consensus.cc:729] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2dc7e8c2808471fb5a9a951c70d5c2a, State: Initialized, Role: LEARNER
I20240502 13:59:52.750808 19809 consensus_queue.cc:260] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 13:59:52.751807 19809 ts_tablet_manager.cc:1398] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 13:59:52.752715 17588 tablet_copy_service.cc:342] P 7c2e9843dbba4851ad72d2d4a8b9886b: Request end of tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-30459a30f0de47ba86791dd018538bce received from {username='slave'} at 127.16.217.7:41287
I20240502 13:59:52.752813 17588 tablet_copy_service.cc:433] P 7c2e9843dbba4851ad72d2d4a8b9886b: ending tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-30459a30f0de47ba86791dd018538bce on tablet 30459a30f0de47ba86791dd018538bce with peer d2dc7e8c2808471fb5a9a951c70d5c2a
W20240502 13:59:52.773118 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:52.782516 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:52.856791 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 13:59:52.857450 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:52.858659 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240502 13:59:52.894292 19087 raft_consensus.cc:1203] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 13:59:52.983367 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:53.119943 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:53.155262 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:53.207073 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 13:59:53.292186 17633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 13:59:53.292908 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 13:59:53.323012 19885 raft_consensus.cc:1050] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b: attempting to promote NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a to VOTER
I20240502 13:59:53.323437 19885 consensus_queue.cc:237] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 13:59:53.324782 17697 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 13:59:53.325171 19886 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:53.325237 19087 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 13:59:53.325323 17502 consensus_peers.cc:485] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.325588 19886 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:53.328184 19888 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.328747 19087 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.331561 17287 catalog_manager.cc:5484] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 5 to 6, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.335095 17697 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.337805 17566 consensus_queue.cc:237] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 13:59:53.338819 19087 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:53.339160 19886 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:53.340529 19834 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.341019 19087 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.428532 17698 raft_consensus.cc:1261] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:53.429526 19834 consensus_queue.cc:1035] T 30459a30f0de47ba86791dd018538bce P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:53.429651 17280 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30459a30f0de47ba86791dd018538bce with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 13:59:53.430435 17287 catalog_manager.cc:5484] T 30459a30f0de47ba86791dd018538bce P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 13:59:53.430470 17698 raft_consensus.cc:2932] T 30459a30f0de47ba86791dd018538bce P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
W20240502 13:59:53.483654 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:53.517765 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:53.641599 17566 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 13:59:53.642925 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 13:59:53.643369 19886 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 13:59:53.643539 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.645184 19834 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } }
I20240502 13:59:53.645612 17698 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } }
I20240502 13:59:53.727084 17291 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:53.730840 17566 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 13:59:53.732683 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 13:59:53.732815 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.733187 19885 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 13:59:53.734948 19885 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
W20240502 13:59:53.736799 17503 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713): Couldn't send request to peer 3305e392697b48a8ad63900628ce4c27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7f702fa6854711b6afbebd76662177. This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:53.737025 17698 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 13:59:53.828878 17280 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2d7f702fa6854711b6afbebd76662177 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 13:59:53.829778 17291 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 4 to 5, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 13:59:54.114071 19923 ts_tablet_manager.cc:902] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Initiating tablet copy from peer 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 13:59:54.114806 19923 tablet_copy_client.cc:308] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:42667
I20240502 13:59:54.118666 17588 tablet_copy_service.cc:140] P 7c2e9843dbba4851ad72d2d4a8b9886b: Received BeginTabletCopySession request for tablet 2d7f702fa6854711b6afbebd76662177 from peer 3305e392697b48a8ad63900628ce4c27 ({username='slave'} at 127.16.217.5:38661)
I20240502 13:59:54.118798 17588 tablet_copy_service.cc:161] P 7c2e9843dbba4851ad72d2d4a8b9886b: Beginning new tablet copy session on tablet 2d7f702fa6854711b6afbebd76662177 from peer 3305e392697b48a8ad63900628ce4c27 at {username='slave'} at 127.16.217.5:38661: session id = 3305e392697b48a8ad63900628ce4c27-2d7f702fa6854711b6afbebd76662177
I20240502 13:59:54.120074 17588 tablet_copy_source_session.cc:205] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:54.123420 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7f702fa6854711b6afbebd76662177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:54.128293 19923 tablet_copy_client.cc:783] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:54.128502 19923 tablet_copy_client.cc:647] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:54.129674 19926 tablet_copy_client.cc:1002] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Injecting latency into file download: 1500
W20240502 13:59:54.575328 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:54.711563 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 13:59:54.870160 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.402798 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.431169 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.476938 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.626499 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 13:59:55.627156 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:55.630060 19923 tablet_copy_client.cc:523] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:55.631968 19923 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Bootstrap starting.
I20240502 13:59:55.637722 19923 log.cc:824] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:55.656697 19923 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 13:59:55.657040 19923 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Bootstrap complete.
I20240502 13:59:55.657332 19923 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240502 13:59:55.659181 19923 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 13:59:55.659395 19923 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3305e392697b48a8ad63900628ce4c27, State: Initialized, Role: LEARNER
I20240502 13:59:55.659709 19923 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 13:59:55.660552 19923 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 13:59:55.661335 17588 tablet_copy_service.cc:342] P 7c2e9843dbba4851ad72d2d4a8b9886b: Request end of tablet copy session 3305e392697b48a8ad63900628ce4c27-2d7f702fa6854711b6afbebd76662177 received from {username='slave'} at 127.16.217.5:38661
I20240502 13:59:55.661424 17588 tablet_copy_service.cc:433] P 7c2e9843dbba4851ad72d2d4a8b9886b: ending tablet copy session 3305e392697b48a8ad63900628ce4c27-2d7f702fa6854711b6afbebd76662177 on tablet 2d7f702fa6854711b6afbebd76662177 with peer 3305e392697b48a8ad63900628ce4c27
W20240502 13:59:55.703054 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.733191 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:55.768581 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:56.045468 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 13:59:56.192054 18825 raft_consensus.cc:1203] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 13:59:56.247313 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:56.658641 19962 raft_consensus.cc:1050] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: attempting to promote NON_VOTER 3305e392697b48a8ad63900628ce4c27 to VOTER
I20240502 13:59:56.659025 19962 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 13:59:56.660369 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 13:59:56.660480 18825 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 13:59:56.660818 20010 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:56.660944 20012 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 13:59:56.661154 17502 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:56.663583 20010 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.664047 18825 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.665491 17698 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.665963 17291 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 5 to 6, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 13:59:56.668978 17566 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 13:59:56.669981 18825 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:56.670060 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:56.670321 20012 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:56.670512 20009 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:56.671882 20010 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.672227 18825 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.672261 17698 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 13:59:56.673723 17280 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d7f702fa6854711b6afbebd76662177 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 13:59:56.674132 17287 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 reported cstate change: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
W20240502 13:59:56.677131 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2d7f702fa6854711b6afbebd76662177: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 13:59:57.033927 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 13:59:57.035295 17566 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 13:59:57.035602 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:57.035631 20035 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 13:59:57.037258 19973 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } }
I20240502 13:59:57.037642 17566 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } }
I20240502 13:59:57.039925 17288 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } }
I20240502 13:59:57.042896 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 13:59:57.044682 17566 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 13:59:57.045370 20037 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 13:59:57.045634 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:57.046974 20035 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 13:59:57.047508 17566 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 13:59:57.048225 17281 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a67fbce1caca46fa818f4e5d90a15305 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 13:59:57.049304 17288 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 4 to 5, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 13:59:57.052137 17763 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031): Couldn't send request to peer c4a5c8f4d3bb4689a4e38e1c743b1c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67fbce1caca46fa818f4e5d90a15305. This is attempt 1: this message will repeat every 5th retry.
W20240502 13:59:57.181667 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 13:59:57.272361 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 13:59:57.572338 20056 ts_tablet_manager.cc:902] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 13:59:57.572968 20056 tablet_copy_client.cc:308] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 13:59:57.576229 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet a67fbce1caca46fa818f4e5d90a15305 from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e ({username='slave'} at 127.16.217.8:48885)
I20240502 13:59:57.576323 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet a67fbce1caca46fa818f4e5d90a15305 from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e at {username='slave'} at 127.16.217.8:48885: session id = c4a5c8f4d3bb4689a4e38e1c743b1c0e-a67fbce1caca46fa818f4e5d90a15305
I20240502 13:59:57.577059 17850 tablet_copy_source_session.cc:205] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 13:59:57.579789 20056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 13:59:57.585038 20056 tablet_copy_client.cc:783] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 0 data blocks...
I20240502 13:59:57.585247 20056 tablet_copy_client.cc:647] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 1 WAL segments...
I20240502 13:59:57.586382 20060 tablet_copy_client.cc:1002] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Injecting latency into file download: 1500
W20240502 13:59:57.696565 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:57.702493 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:57.878461 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 13:59:57.998241 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 13:59:57.998807 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:58.236371 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:58.277915 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:58.294006 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:58.363494 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 13:59:58.722208 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:58.848498 17633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 13:59:59.086833 20056 tablet_copy_client.cc:523] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 13:59:59.091162 20056 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap starting.
I20240502 13:59:59.096745 20056 log.cc:824] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Log is configured to *not* fsync() on all Append() calls
I20240502 13:59:59.115634 20056 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 13:59:59.115927 20056 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap complete.
I20240502 13:59:59.116170 20056 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent bootstrapping tablet: real 0.025s user 0.022s sys 0.000s
I20240502 13:59:59.117969 20056 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 13:59:59.118165 20056 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a5c8f4d3bb4689a4e38e1c743b1c0e, State: Initialized, Role: LEARNER
I20240502 13:59:59.118429 20056 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 13:59:59.119174 20056 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240502 13:59:59.119791 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-a67fbce1caca46fa818f4e5d90a15305 received from {username='slave'} at 127.16.217.8:48885
I20240502 13:59:59.119882 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-a67fbce1caca46fa818f4e5d90a15305 on tablet a67fbce1caca46fa818f4e5d90a15305 with peer c4a5c8f4d3bb4689a4e38e1c743b1c0e
W20240502 13:59:59.322813 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 13:59:59.356732 19218 raft_consensus.cc:1203] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 13:59:59.528643 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 13:59:59.792975 20130 raft_consensus.cc:1050] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e to VOTER
I20240502 13:59:59.793314 20130 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 13:59:59.794638 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 13:59:59.795032 20132 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:59.795086 17567 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 13:59:59.795292 17764 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 13:59:59.795518 20133 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 13:59:59.798146 20132 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.798601 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.800798 17288 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e reported cstate change: config changed from index 5 to 6, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.801908 17567 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.805166 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 13:59:59.806031 17567 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:59.806114 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 13:59:59.806396 20133 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:59.806511 20130 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 13:59:59.807976 20132 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.808741 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.808743 17567 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 13:59:59.810158 17291 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 13:59:59.810283 17281 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a67fbce1caca46fa818f4e5d90a15305 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 13:59:59.813221 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a67fbce1caca46fa818f4e5d90a15305: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 13:59:59.861101 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 13:59:59.885260 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:00:00.138890 17698 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 14:00:00.140187 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:00.140702 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:00.140846 20154 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:00.142473 19711 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } }
I20240502 14:00:00.142863 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } }
I20240502 14:00:00.144697 17288 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } }
I20240502 14:00:00.147626 17698 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:00.149128 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:00.149828 19711 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:00:00.150190 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:00.151284 20154 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:00.152472 17279 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4a240577417f4ae1812bac08ba71861b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 14:00:00.152863 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:00.153198 17288 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 4 to 5, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:00.153491 17635 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157): Couldn't send request to peer 73731152a18648aab3b1a720dabcc26b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a240577417f4ae1812bac08ba71861b. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:00.508845 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:00:00.602993 20175 ts_tablet_manager.cc:902] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Initiating tablet copy from peer b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 14:00:00.603459 20175 tablet_copy_client.cc:308] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:42979
I20240502 14:00:00.606784 17719 tablet_copy_service.cc:140] P b0da919a0055405e8ef4dfd545e3bd23: Received BeginTabletCopySession request for tablet 4a240577417f4ae1812bac08ba71861b from peer 73731152a18648aab3b1a720dabcc26b ({username='slave'} at 127.16.217.6:54231)
I20240502 14:00:00.606902 17719 tablet_copy_service.cc:161] P b0da919a0055405e8ef4dfd545e3bd23: Beginning new tablet copy session on tablet 4a240577417f4ae1812bac08ba71861b from peer 73731152a18648aab3b1a720dabcc26b at {username='slave'} at 127.16.217.6:54231: session id = 73731152a18648aab3b1a720dabcc26b-4a240577417f4ae1812bac08ba71861b
I20240502 14:00:00.607978 17719 tablet_copy_source_session.cc:205] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:00.609079 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a240577417f4ae1812bac08ba71861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:00.610668 20175 tablet_copy_client.cc:783] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:00.610801 20175 tablet_copy_client.cc:647] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:00.612085 20179 tablet_copy_client.cc:1002] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:00.751677 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:00.773593 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:00.941825 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240502 14:00:00.942466 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a67fbce1caca46fa818f4e5d90a15305: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:00.957630 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:01.829216 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:00:01.987032 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a67fbce1caca46fa818f4e5d90a15305: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:02.112453 20175 tablet_copy_client.cc:523] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:02.113826 20175 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Bootstrap starting.
I20240502 14:00:02.129671 20175 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:02.129930 20175 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Bootstrap complete.
I20240502 14:00:02.130105 20175 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.000s
I20240502 14:00:02.130645 20175 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:02.130751 20175 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Initialized, Role: LEARNER
I20240502 14:00:02.130836 20175 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:02.131145 20175 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:02.131738 17719 tablet_copy_service.cc:342] P b0da919a0055405e8ef4dfd545e3bd23: Request end of tablet copy session 73731152a18648aab3b1a720dabcc26b-4a240577417f4ae1812bac08ba71861b received from {username='slave'} at 127.16.217.6:54231
I20240502 14:00:02.131822 17719 tablet_copy_service.cc:433] P b0da919a0055405e8ef4dfd545e3bd23: ending tablet copy session 73731152a18648aab3b1a720dabcc26b-4a240577417f4ae1812bac08ba71861b on tablet 4a240577417f4ae1812bac08ba71861b with peer 73731152a18648aab3b1a720dabcc26b
W20240502 14:00:02.203086 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:02.442342 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:00:02.478236 18956 raft_consensus.cc:1203] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 14:00:02.735677 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:00:02.862826 20194 raft_consensus.cc:1050] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: attempting to promote NON_VOTER 73731152a18648aab3b1a720dabcc26b to VOTER
I20240502 14:00:02.863291 20194 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:02.864715 18956 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:02.864828 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:02.865211 20254 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:02.865430 20252 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:02.865808 17633 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:02.867380 20252 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.867807 18956 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.868986 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.870113 17287 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 5 to 6, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.873529 17698 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:02.874519 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:02.874585 18956 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:02.874890 20255 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:02.875020 20252 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:02.876403 20252 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.876972 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.876976 18956 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:02.878558 17279 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a240577417f4ae1812bac08ba71861b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:00:02.878751 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240502 14:00:02.879244 17288 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 6 to 7, VOTER b36b507eb389473b8438e907237a63f9 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:00:03.071250 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:00:03.071825 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:03.080729 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a240577417f4ae1812bac08ba71861b: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:03.263439 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:00:03.288393 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.292701 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.293176 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:03.296901 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.304148 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.306002 17809 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:00:03.328860 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:00:03.366434 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:00:03.401489 18936 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.404659 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.404712 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.407191 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:03.415771 19067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:03.424454 17679 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:03.429391 17809 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 14:00:03.450482 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:00:03.465519 17698 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:03.466825 17567 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:03.467049 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:03.467317 20194 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:03.467542 20352 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:03.469384 20353 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:03.469871 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:03.471936 17287 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:03.472183 17635 consensus_peers.cc:485] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 -> Peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123): Couldn't send request to peer d2dc7e8c2808471fb5a9a951c70d5c2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda5fbedb8b54b56ab058434f4eb4395. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:03.472095 17567 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:03.480151 17698 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:03.481340 18956 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:03.481362 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:03.481686 20353 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:03.481835 20352 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:03.483587 20353 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:03.484472 18956 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:03.484462 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
W20240502 14:00:03.484597 17632 consensus_peers.cc:485] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 -> Peer c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031): Couldn't send request to peer c4a5c8f4d3bb4689a4e38e1c743b1c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a240577417f4ae1812bac08ba71861b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:03.486440 17294 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:03.899528 20387 ts_tablet_manager.cc:902] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Initiating tablet copy from peer b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 14:00:03.900009 20387 tablet_copy_client.cc:308] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:42979
I20240502 14:00:03.903004 17719 tablet_copy_service.cc:140] P b0da919a0055405e8ef4dfd545e3bd23: Received BeginTabletCopySession request for tablet dda5fbedb8b54b56ab058434f4eb4395 from peer d2dc7e8c2808471fb5a9a951c70d5c2a ({username='slave'} at 127.16.217.7:48659)
I20240502 14:00:03.903095 17719 tablet_copy_service.cc:161] P b0da919a0055405e8ef4dfd545e3bd23: Beginning new tablet copy session on tablet dda5fbedb8b54b56ab058434f4eb4395 from peer d2dc7e8c2808471fb5a9a951c70d5c2a at {username='slave'} at 127.16.217.7:48659: session id = d2dc7e8c2808471fb5a9a951c70d5c2a-dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:03.903760 17719 tablet_copy_source_session.cc:205] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:03.904522 20387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:03.906143 20387 tablet_copy_client.cc:783] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:03.906302 20387 tablet_copy_client.cc:647] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:03.907351 20391 tablet_copy_client.cc:1002] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Injecting latency into file download: 1500
I20240502 14:00:03.912246 20393 ts_tablet_manager.cc:902] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Initiating tablet copy from peer b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979)
I20240502 14:00:03.912662 20393 tablet_copy_client.cc:308] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:42979
I20240502 14:00:03.915879 17719 tablet_copy_service.cc:140] P b0da919a0055405e8ef4dfd545e3bd23: Received BeginTabletCopySession request for tablet 4a240577417f4ae1812bac08ba71861b from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e ({username='slave'} at 127.16.217.8:46033)
I20240502 14:00:03.915953 17719 tablet_copy_service.cc:161] P b0da919a0055405e8ef4dfd545e3bd23: Beginning new tablet copy session on tablet 4a240577417f4ae1812bac08ba71861b from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e at {username='slave'} at 127.16.217.8:46033: session id = c4a5c8f4d3bb4689a4e38e1c743b1c0e-4a240577417f4ae1812bac08ba71861b
I20240502 14:00:03.916652 17719 tablet_copy_source_session.cc:205] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:03.917377 20393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a240577417f4ae1812bac08ba71861b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:03.918829 20393 tablet_copy_client.cc:783] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:03.918957 20393 tablet_copy_client.cc:647] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:03.919853 20397 tablet_copy_client.cc:1002] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:04.054680 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:00:04.521538 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:04.835656 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:05.069757 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2d7f702fa6854711b6afbebd76662177: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:05.077298 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:05.407613 20387 tablet_copy_client.cc:523] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:05.409207 20387 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap starting.
I20240502 14:00:05.420137 20393 tablet_copy_client.cc:523] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:05.421345 20393 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap starting.
I20240502 14:00:05.427104 20387 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:05.427350 20387 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap complete.
I20240502 14:00:05.427492 20387 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 14:00:05.427995 20387 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:05.428108 20387 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2dc7e8c2808471fb5a9a951c70d5c2a, State: Initialized, Role: LEARNER
I20240502 14:00:05.428239 20387 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:05.428566 20387 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:05.429067 17719 tablet_copy_service.cc:342] P b0da919a0055405e8ef4dfd545e3bd23: Request end of tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-dda5fbedb8b54b56ab058434f4eb4395 received from {username='slave'} at 127.16.217.7:48659
I20240502 14:00:05.429159 17719 tablet_copy_service.cc:433] P b0da919a0055405e8ef4dfd545e3bd23: ending tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-dda5fbedb8b54b56ab058434f4eb4395 on tablet dda5fbedb8b54b56ab058434f4eb4395 with peer d2dc7e8c2808471fb5a9a951c70d5c2a
I20240502 14:00:05.439424 20393 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:05.439646 20393 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap complete.
I20240502 14:00:05.439790 20393 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20240502 14:00:05.440192 20393 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:05.440330 20393 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a5c8f4d3bb4689a4e38e1c743b1c0e, State: Initialized, Role: LEARNER
I20240502 14:00:05.440433 20393 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:05.440682 20393 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 14:00:05.441169 17719 tablet_copy_service.cc:342] P b0da919a0055405e8ef4dfd545e3bd23: Request end of tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-4a240577417f4ae1812bac08ba71861b received from {username='slave'} at 127.16.217.8:46033
I20240502 14:00:05.441273 17719 tablet_copy_service.cc:433] P b0da919a0055405e8ef4dfd545e3bd23: ending tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-4a240577417f4ae1812bac08ba71861b on tablet 4a240577417f4ae1812bac08ba71861b with peer c4a5c8f4d3bb4689a4e38e1c743b1c0e
W20240502 14:00:05.575560 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:00:05.703590 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:05.751911 19087 raft_consensus.cc:1203] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:05.809417 19218 raft_consensus.cc:1203] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:00:05.992769 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:06.342698 20532 raft_consensus.cc:1050] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23: attempting to promote NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e to VOTER
I20240502 14:00:06.343215 20532 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:06.344830 19218 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:06.344897 18956 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:06.344898 17567 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:06.345206 20534 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:06.345423 20531 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:06.345628 20532 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:06.347442 20534 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.347900 19218 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.348057 17567 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.349958 18956 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.350255 17287 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:06.354069 17698 consensus_queue.cc:237] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:06.355212 19218 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:06.355504 18956 raft_consensus.cc:1261] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:06.355640 20534 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:06.356078 20432 consensus_queue.cc:1035] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:06.357172 20534 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.357815 19218 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.357975 18956 raft_consensus.cc:2932] T 4a240577417f4ae1812bac08ba71861b P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:06.358893 17279 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4a240577417f4ae1812bac08ba71861b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:06.359683 17294 catalog_manager.cc:5484] T 4a240577417f4ae1812bac08ba71861b P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:06.361654 20532 raft_consensus.cc:1050] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: attempting to promote NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a to VOTER
I20240502 14:00:06.361951 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a240577417f4ae1812bac08ba71861b with delete_type TABLET_DATA_TOMBSTONED (TS 7c2e9843dbba4851ad72d2d4a8b9886b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54388
I20240502 14:00:06.362001 20532 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:06.362762 20544 tablet_replica.cc:330] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: stopping tablet replica
I20240502 14:00:06.363322 20544 raft_consensus.cc:2229] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:06.363456 19087 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:06.363479 17567 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:06.363624 20544 raft_consensus.cc:2258] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:06.363706 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer b0da919a0055405e8ef4dfd545e3bd23: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:06.364069 20531 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:06.364264 20532 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:06.364416 20432 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:06.365567 20544 ts_tablet_manager.cc:1875] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:06.367043 20532 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:06.367513 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:06.367931 20544 ts_tablet_manager.cc:1888] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:00:06.368006 20544 log.cc:1196] T 4a240577417f4ae1812bac08ba71861b P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a240577417f4ae1812bac08ba71861b
I20240502 14:00:06.369414 19087 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:06.369652 17290 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b0da919a0055405e8ef4dfd545e3bd23" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:06.370875 17280 catalog_manager.cc:4830] TS 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667): tablet 4a240577417f4ae1812bac08ba71861b (table rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]) successfully deleted
I20240502 14:00:06.373960 17567 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:06.476110 17698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395"
dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57282
I20240502 14:00:06.476296 17698 raft_consensus.cc:595] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:06.483259 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:06.484752 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:06.484752 17567 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 14:00:06.485069 17766 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157): Couldn't send request to peer 73731152a18648aab3b1a720dabcc26b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67fbce1caca46fa818f4e5d90a15305. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:06.485261 20572 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:06.485440 20520 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:06.487036 20219 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:06.487448 17567 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:06.489776 17290 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:06.492831 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:06.497443 17829 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:06.498687 17567 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:06.499042 18956 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:06.499027 20520 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:06.499594 20520 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:00:06.499733 17763 consensus_peers.cc:485] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 -> Peer c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031): Couldn't send request to peer c4a5c8f4d3bb4689a4e38e1c743b1c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b05abb189494b6592787e2f17b4c33a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:06.500679 20219 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:06.501101 17567 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:06.501749 18956 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:06.503715 17291 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:06.518486 17698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395"
dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57292
I20240502 14:00:06.518602 17698 raft_consensus.cc:595] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:06.714985 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:00:06.738852 17698 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395"
dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57312
I20240502 14:00:06.738988 17698 raft_consensus.cc:595] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:06.798964 20432 raft_consensus.cc:979] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: : Instructing follower d2dc7e8c2808471fb5a9a951c70d5c2a to start an election
I20240502 14:00:06.799088 20432 raft_consensus.cc:1067] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Signalling peer d2dc7e8c2808471fb5a9a951c70d5c2a to start an election
I20240502 14:00:06.799685 19087 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395"
dest_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
from {username='slave'} at 127.16.217.3:57661
I20240502 14:00:06.799855 19087 raft_consensus.cc:482] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:06.799961 19087 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:06.800935 19087 raft_consensus.cc:504] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:06.801959 19087 leader_election.cc:288] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [CANDIDATE]: Term 2 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 14:00:06.802268 17698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b0da919a0055405e8ef4dfd545e3bd23"
I20240502 14:00:06.802385 17698 raft_consensus.cc:3032] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:06.802426 17698 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b0da919a0055405e8ef4dfd545e3bd23, State: Running, Role: LEADER
I20240502 14:00:06.802419 17567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 14:00:06.802553 17567 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:06.802573 17698 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:06.802784 17698 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:06.803603 17567 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dc7e8c2808471fb5a9a951c70d5c2a in term 2.
I20240502 14:00:06.803666 17698 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dc7e8c2808471fb5a9a951c70d5c2a in term 2.
I20240502 14:00:06.804005 19020 leader_election.cc:302] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7c2e9843dbba4851ad72d2d4a8b9886b, b0da919a0055405e8ef4dfd545e3bd23, d2dc7e8c2808471fb5a9a951c70d5c2a; no voters:
I20240502 14:00:06.804248 20549 raft_consensus.cc:2781] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:06.804456 20549 raft_consensus.cc:686] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Becoming Leader. State: Replica: d2dc7e8c2808471fb5a9a951c70d5c2a, State: Running, Role: LEADER
I20240502 14:00:06.804777 20549 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
W20240502 14:00:06.805029 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:06.806056 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dda5fbedb8b54b56ab058434f4eb4395" candidate_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 14:00:06.806185 17829 raft_consensus.cc:3037] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:06.807127 17829 raft_consensus.cc:2454] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2dc7e8c2808471fb5a9a951c70d5c2a in term 2.
I20240502 14:00:06.807519 17287 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: term changed from 1 to 2, leader changed from b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) to d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7). New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:06.869675 20625 ts_tablet_manager.cc:902] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 14:00:06.870165 20625 tablet_copy_client.cc:308] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 14:00:06.870669 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet a67fbce1caca46fa818f4e5d90a15305 from peer 73731152a18648aab3b1a720dabcc26b ({username='slave'} at 127.16.217.6:39333)
I20240502 14:00:06.870788 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet a67fbce1caca46fa818f4e5d90a15305 from peer 73731152a18648aab3b1a720dabcc26b at {username='slave'} at 127.16.217.6:39333: session id = 73731152a18648aab3b1a720dabcc26b-a67fbce1caca46fa818f4e5d90a15305
I20240502 14:00:06.871520 17850 tablet_copy_source_session.cc:205] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:06.872431 20625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:06.874169 20625 tablet_copy_client.cc:783] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:06.874317 20625 tablet_copy_client.cc:647] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:06.875342 20629 tablet_copy_client.cc:1002] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:06.927795 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240502 14:00:06.928699 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:06.955022 20643 ts_tablet_manager.cc:902] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 14:00:06.955574 20643 tablet_copy_client.cc:308] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 14:00:06.956128 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet 5b05abb189494b6592787e2f17b4c33a from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e ({username='slave'} at 127.16.217.8:48885)
I20240502 14:00:06.956238 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet 5b05abb189494b6592787e2f17b4c33a from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e at {username='slave'} at 127.16.217.8:48885: session id = c4a5c8f4d3bb4689a4e38e1c743b1c0e-5b05abb189494b6592787e2f17b4c33a
I20240502 14:00:06.957230 17850 tablet_copy_source_session.cc:205] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:06.958086 20643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:06.960158 20643 tablet_copy_client.cc:783] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:06.960397 20643 tablet_copy_client.cc:647] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:06.961727 20647 tablet_copy_client.cc:1002] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:07.163993 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:07.224025 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:00:07.224596 20549 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:07.264499 17698 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:00:07.264989 20549 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:07.268852 17567 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:00:07.269485 20624 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:00:07.730067 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:07.824968 19087 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:07.826157 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:07.826503 17567 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:00:07.826542 20711 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:07.826995 20711 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:07.828763 20712 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:07.829344 17567 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:07.829404 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:07.830204 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dda5fbedb8b54b56ab058434f4eb4395 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:07.832152 17294 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 5 to 7, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:07.835386 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 with delete_type TABLET_DATA_TOMBSTONED (TS b0da919a0055405e8ef4dfd545e3bd23 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60834
I20240502 14:00:07.835862 20729 tablet_replica.cc:330] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: stopping tablet replica
I20240502 14:00:07.836119 20729 raft_consensus.cc:2229] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:07.836267 20729 raft_consensus.cc:2258] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:07.837255 20729 ts_tablet_manager.cc:1875] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:07.838979 20729 ts_tablet_manager.cc:1888] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:00:07.839059 20729 log.cc:1196] T dda5fbedb8b54b56ab058434f4eb4395 P b0da919a0055405e8ef4dfd545e3bd23: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:07.839596 17279 catalog_manager.cc:4830] TS b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979): tablet dda5fbedb8b54b56ab058434f4eb4395 (table rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]) successfully deleted
W20240502 14:00:07.894789 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:00:07.895553 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:08.057729 17829 consensus_queue.cc:237] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:08.059202 17567 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:08.059283 17698 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:08.059762 20668 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:08.060425 20668 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:08.062237 20219 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:08.064327 17567 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
W20240502 14:00:08.064671 17765 consensus_peers.cc:485] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 -> Peer 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713): Couldn't send request to peer 3305e392697b48a8ad63900628ce4c27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb05a5ed865346528523ae99a8d3882b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:08.064867 17287 catalog_manager.cc:5484] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index -1 to 4, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:08.065522 17698 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
W20240502 14:00:08.174063 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a67fbce1caca46fa818f4e5d90a15305: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:08.328610 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:00:08.375706 20625 tablet_copy_client.cc:523] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.377007 20625 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Bootstrap starting.
I20240502 14:00:08.395666 20625 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.395985 20625 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Bootstrap complete.
I20240502 14:00:08.396179 20625 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20240502 14:00:08.396696 20625 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:08.396871 20625 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Initialized, Role: LEARNER
I20240502 14:00:08.397010 20625 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:08.397384 20625 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
W20240502 14:00:08.397595 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:08.397929 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session 73731152a18648aab3b1a720dabcc26b-a67fbce1caca46fa818f4e5d90a15305 received from {username='slave'} at 127.16.217.6:39333
I20240502 14:00:08.398025 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session 73731152a18648aab3b1a720dabcc26b-a67fbce1caca46fa818f4e5d90a15305 on tablet a67fbce1caca46fa818f4e5d90a15305 with peer 73731152a18648aab3b1a720dabcc26b
I20240502 14:00:08.407092 18956 raft_consensus.cc:1203] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:00:08.457334 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:08.462031 20643 tablet_copy_client.cc:523] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:08.463594 20643 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap starting.
I20240502 14:00:08.481087 20643 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:08.481344 20643 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap complete.
I20240502 14:00:08.481477 20643 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20240502 14:00:08.481858 20643 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:08.481969 20643 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a5c8f4d3bb4689a4e38e1c743b1c0e, State: Initialized, Role: LEARNER
I20240502 14:00:08.482043 20643 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:08.482378 20643 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:08.483011 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-5b05abb189494b6592787e2f17b4c33a received from {username='slave'} at 127.16.217.8:48885
I20240502 14:00:08.483115 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-5b05abb189494b6592787e2f17b4c33a on tablet 5b05abb189494b6592787e2f17b4c33a with peer c4a5c8f4d3bb4689a4e38e1c743b1c0e
I20240502 14:00:08.618809 20806 ts_tablet_manager.cc:902] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 14:00:08.619278 20806 tablet_copy_client.cc:308] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 14:00:08.622514 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet bb05a5ed865346528523ae99a8d3882b from peer 3305e392697b48a8ad63900628ce4c27 ({username='slave'} at 127.16.217.5:36523)
I20240502 14:00:08.622610 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet bb05a5ed865346528523ae99a8d3882b from peer 3305e392697b48a8ad63900628ce4c27 at {username='slave'} at 127.16.217.5:36523: session id = 3305e392697b48a8ad63900628ce4c27-bb05a5ed865346528523ae99a8d3882b
I20240502 14:00:08.623386 17850 tablet_copy_source_session.cc:205] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:08.624169 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb05a5ed865346528523ae99a8d3882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:08.626238 20806 tablet_copy_client.cc:783] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:08.626387 20806 tablet_copy_client.cc:647] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:08.627341 20810 tablet_copy_client.cc:1002] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: tablet copy: Injecting latency into file download: 1500
I20240502 14:00:08.872186 20669 raft_consensus.cc:1050] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER 73731152a18648aab3b1a720dabcc26b to VOTER
I20240502 14:00:08.872572 20669 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:08.873746 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:08.873762 18956 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:08.873857 17566 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:08.874078 20669 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:08.874238 20829 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:08.874423 20827 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:08.876488 20827 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.877115 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.877341 17566 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.879601 18956 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.879657 17291 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.884135 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:08.885100 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:08.885133 18956 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:08.885450 20219 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:08.885583 20827 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:08.886997 20829 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.887312 18956 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.887441 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:08.888785 17281 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a67fbce1caca46fa818f4e5d90a15305 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:08.889497 17290 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:08.891477 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fbce1caca46fa818f4e5d90a15305 with delete_type TABLET_DATA_TOMBSTONED (TS 7c2e9843dbba4851ad72d2d4a8b9886b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54388
I20240502 14:00:08.891786 20836 tablet_replica.cc:330] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: stopping tablet replica
I20240502 14:00:08.891968 20836 raft_consensus.cc:2229] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:08.892205 20836 raft_consensus.cc:2258] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:08.892827 20836 ts_tablet_manager.cc:1875] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:08.894649 20836 ts_tablet_manager.cc:1888] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:00:08.894716 20836 log.cc:1196] T a67fbce1caca46fa818f4e5d90a15305 P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/a67fbce1caca46fa818f4e5d90a15305
I20240502 14:00:08.895133 17280 catalog_manager.cc:4830] TS 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667): tablet a67fbce1caca46fa818f4e5d90a15305 (table rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]) successfully deleted
I20240502 14:00:09.035249 19218 raft_consensus.cc:1203] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:00:09.373402 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:09.414958 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:00:09.476143 20219 raft_consensus.cc:1050] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e to VOTER
I20240502 14:00:09.476645 20219 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:09.477909 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:09.478078 18956 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:09.478188 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:09.478543 20883 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:09.478706 20827 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:09.478930 20669 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:09.481117 20669 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:09.481570 19218 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:09.481745 18956 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:09.483780 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:09.483879 17294 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e reported cstate change: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:09.599813 17829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a"
dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43078
I20240502 14:00:09.599995 17829 raft_consensus.cc:595] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:09.659636 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:00:09.814039 17829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a"
dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59398
I20240502 14:00:09.814152 17829 raft_consensus.cc:595] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:09.972106 20219 raft_consensus.cc:979] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: : Instructing follower 73731152a18648aab3b1a720dabcc26b to start an election
I20240502 14:00:09.972218 20219 raft_consensus.cc:1067] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Signalling peer 73731152a18648aab3b1a720dabcc26b to start an election
I20240502 14:00:09.972754 18956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a"
dest_uuid: "73731152a18648aab3b1a720dabcc26b"
from {username='slave'} at 127.16.217.4:47771
I20240502 14:00:09.972916 18956 raft_consensus.cc:482] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:09.972997 18956 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.973750 18956 raft_consensus.cc:504] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:09.974706 18956 leader_election.cc:288] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [CANDIDATE]: Term 2 election: Requested vote from peers 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667), e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769), c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031)
I20240502 14:00:09.975365 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 14:00:09.975476 17829 raft_consensus.cc:3032] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:09.975517 17829 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 14:00:09.975677 17829 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:09.975886 17829 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.976756 17829 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
I20240502 14:00:09.978912 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b"
I20240502 14:00:09.979058 17566 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.979943 17566 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
I20240502 14:00:09.980394 18890 leader_election.cc:302] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 73731152a18648aab3b1a720dabcc26b, 7c2e9843dbba4851ad72d2d4a8b9886b, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 14:00:09.980507 19218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b05abb189494b6592787e2f17b4c33a" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
I20240502 14:00:09.980643 20887 raft_consensus.cc:2781] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:09.980703 19218 raft_consensus.cc:3037] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:09.980856 20887 raft_consensus.cc:686] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Becoming Leader. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Running, Role: LEADER
I20240502 14:00:09.981122 20887 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:09.981654 19218 raft_consensus.cc:2454] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
I20240502 14:00:09.983592 17287 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: term changed from 1 to 2, leader changed from e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) to 73731152a18648aab3b1a720dabcc26b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:10.069730 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:10.127671 20806 tablet_copy_client.cc:523] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:10.129307 20806 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Bootstrap starting.
I20240502 14:00:10.144390 20806 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:10.144657 20806 tablet_bootstrap.cc:492] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Bootstrap complete.
I20240502 14:00:10.144812 20806 ts_tablet_manager.cc:1367] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.000s
I20240502 14:00:10.145305 20806 raft_consensus.cc:348] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:10.145401 20806 raft_consensus.cc:729] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3305e392697b48a8ad63900628ce4c27, State: Initialized, Role: LEARNER
I20240502 14:00:10.145479 20806 consensus_queue.cc:260] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:10.145825 20806 ts_tablet_manager.cc:1398] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:10.146317 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session 3305e392697b48a8ad63900628ce4c27-bb05a5ed865346528523ae99a8d3882b received from {username='slave'} at 127.16.217.5:36523
I20240502 14:00:10.146389 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session 3305e392697b48a8ad63900628ce4c27-bb05a5ed865346528523ae99a8d3882b on tablet bb05a5ed865346528523ae99a8d3882b with peer 3305e392697b48a8ad63900628ce4c27
W20240502 14:00:10.357404 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:00:10.375027 18825 raft_consensus.cc:1203] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:00:10.430042 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:00:10.430521 20887 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:10.467993 17829 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:00:10.468045 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:00:10.468441 20925 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:10.468595 20887 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 14:00:10.475550 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:00:10.478219 18956 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:10.479218 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:00:10.479231 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:00:10.479652 20887 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:10.479825 20966 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:10.481220 20887 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:10.482641 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:10.482641 19218 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:10.484138 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b05abb189494b6592787e2f17b4c33a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:10.484812 17294 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 9 to 11, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:10.487226 17809 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b05abb189494b6592787e2f17b4c33a with delete_type TABLET_DATA_TOMBSTONED (TS e7f378d8d4c745d888ed99454393bb85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:44318
I20240502 14:00:10.487641 20973 tablet_replica.cc:330] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: stopping tablet replica
I20240502 14:00:10.488023 20973 raft_consensus.cc:2229] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:10.488219 20973 raft_consensus.cc:2258] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:10.489218 20973 ts_tablet_manager.cc:1875] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:10.491040 20973 ts_tablet_manager.cc:1888] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 14:00:10.491117 20973 log.cc:1196] T 5b05abb189494b6592787e2f17b4c33a P e7f378d8d4c745d888ed99454393bb85: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b05abb189494b6592787e2f17b4c33a
I20240502 14:00:10.491660 17281 catalog_manager.cc:4830] TS e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769): tablet 5b05abb189494b6592787e2f17b4c33a (table rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]) successfully deleted
W20240502 14:00:10.826046 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:00:10.892961 20969 raft_consensus.cc:1050] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER 3305e392697b48a8ad63900628ce4c27 to VOTER
I20240502 14:00:10.893415 20969 consensus_queue.cc:237] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:10.894615 18825 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:10.894629 17566 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:10.894752 17697 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:10.894974 20994 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:10.895212 20969 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:10.895375 20996 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:10.897261 20996 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.897678 17566 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.898032 18825 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.899958 17290 catalog_manager.cc:5484] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 4 to 5, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.901203 17697 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.904464 17829 consensus_queue.cc:237] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:10.905512 18825 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:10.905577 17566 raft_consensus.cc:1261] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:10.905822 20996 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:10.905951 20994 consensus_queue.cc:1035] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:10.907311 20969 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.907694 17566 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.907807 18825 raft_consensus.cc:2932] T bb05a5ed865346528523ae99a8d3882b P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:10.909194 17281 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb05a5ed865346528523ae99a8d3882b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:10.909286 17291 catalog_manager.cc:5484] T bb05a5ed865346528523ae99a8d3882b P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:10.911509 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet bb05a5ed865346528523ae99a8d3882b with delete_type TABLET_DATA_TOMBSTONED (TS b0da919a0055405e8ef4dfd545e3bd23 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60834
I20240502 14:00:10.911859 21001 tablet_replica.cc:330] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: stopping tablet replica
I20240502 14:00:10.911993 21001 raft_consensus.cc:2229] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:10.912155 21001 raft_consensus.cc:2258] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:10.913148 21001 ts_tablet_manager.cc:1875] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:10.914716 21001 ts_tablet_manager.cc:1888] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:10.914772 21001 log.cc:1196] T bb05a5ed865346528523ae99a8d3882b P b0da919a0055405e8ef4dfd545e3bd23: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb05a5ed865346528523ae99a8d3882b
I20240502 14:00:10.915202 17279 catalog_manager.cc:4830] TS b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979): tablet bb05a5ed865346528523ae99a8d3882b (table rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]) successfully deleted
I20240502 14:00:11.082149 17566 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:11.083531 17697 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:00:11.083591 17829 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:00:11.083770 17504 consensus_peers.cc:485] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123): Couldn't send request to peer d2dc7e8c2808471fb5a9a951c70d5c2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 224b4919644d45dc942fb6654eb54c84. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:11.084000 21014 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:11.084187 21015 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:00:11.085922 20119 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:11.086309 17829 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:11.086371 17697 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:11.088697 17287 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index -1 to 4, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:11.134541 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:11.297940 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a240577417f4ae1812bac08ba71861b: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:11.568523 21035 ts_tablet_manager.cc:902] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Initiating tablet copy from peer 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 14:00:11.569023 21035 tablet_copy_client.cc:308] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:42667
I20240502 14:00:11.569442 17588 tablet_copy_service.cc:140] P 7c2e9843dbba4851ad72d2d4a8b9886b: Received BeginTabletCopySession request for tablet 224b4919644d45dc942fb6654eb54c84 from peer d2dc7e8c2808471fb5a9a951c70d5c2a ({username='slave'} at 127.16.217.7:41287)
I20240502 14:00:11.569540 17588 tablet_copy_service.cc:161] P 7c2e9843dbba4851ad72d2d4a8b9886b: Beginning new tablet copy session on tablet 224b4919644d45dc942fb6654eb54c84 from peer d2dc7e8c2808471fb5a9a951c70d5c2a at {username='slave'} at 127.16.217.7:41287: session id = d2dc7e8c2808471fb5a9a951c70d5c2a-224b4919644d45dc942fb6654eb54c84
I20240502 14:00:11.570240 17588 tablet_copy_source_session.cc:205] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:11.571081 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b4919644d45dc942fb6654eb54c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:11.572628 21035 tablet_copy_client.cc:783] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:11.572760 21035 tablet_copy_client.cc:647] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:11.573702 21039 tablet_copy_client.cc:1002] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:11.712659 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:11.797215 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:00:12.512950 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:12.909385 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:00:12.910017 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:00:13.074118 21035 tablet_copy_client.cc:523] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:13.075563 21035 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap starting.
I20240502 14:00:13.091323 21035 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:13.091571 21035 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap complete.
I20240502 14:00:13.091729 21035 ts_tablet_manager.cc:1367] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20240502 14:00:13.092157 21035 raft_consensus.cc:348] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:13.092303 21035 raft_consensus.cc:729] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2dc7e8c2808471fb5a9a951c70d5c2a, State: Initialized, Role: LEARNER
I20240502 14:00:13.092402 21035 consensus_queue.cc:260] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:13.092643 21035 ts_tablet_manager.cc:1398] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 14:00:13.093282 17588 tablet_copy_service.cc:342] P 7c2e9843dbba4851ad72d2d4a8b9886b: Request end of tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-224b4919644d45dc942fb6654eb54c84 received from {username='slave'} at 127.16.217.7:41287
I20240502 14:00:13.093389 17588 tablet_copy_service.cc:433] P 7c2e9843dbba4851ad72d2d4a8b9886b: ending tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-224b4919644d45dc942fb6654eb54c84 on tablet 224b4919644d45dc942fb6654eb54c84 with peer d2dc7e8c2808471fb5a9a951c70d5c2a
I20240502 14:00:13.159333 19086 raft_consensus.cc:1203] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 14:00:13.211892 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:13.311667 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 14:00:13.312389 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 2d7f702fa6854711b6afbebd76662177: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:13.424585 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:00:13.564854 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 14:00:13.664995 21100 raft_consensus.cc:1050] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: attempting to promote NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a to VOTER
I20240502 14:00:13.665393 21100 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:13.666815 19086 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:13.666878 17829 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:13.666951 17698 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:00:13.667261 21100 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:13.667438 21102 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:13.667582 21103 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:00:13.669243 21103 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.669664 17698 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.669988 17829 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.671105 19086 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.672405 17291 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 4 to 5, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:13.675855 17566 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:13.676873 19086 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:13.676965 17829 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:00:13.677217 21102 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:13.677374 21100 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:00:13.678805 21070 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.679185 19086 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.680387 17280 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 224b4919644d45dc942fb6654eb54c84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:13.680485 17829 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:13.681262 17287 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 5 to 6, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
W20240502 14:00:13.682363 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:00:13.683734 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet 224b4919644d45dc942fb6654eb54c84 with delete_type TABLET_DATA_TOMBSTONED (TS b0da919a0055405e8ef4dfd545e3bd23 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60834
I20240502 14:00:13.684095 21111 tablet_replica.cc:330] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: stopping tablet replica
I20240502 14:00:13.684229 21111 raft_consensus.cc:2229] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:13.684413 21111 raft_consensus.cc:2258] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:13.685056 21111 ts_tablet_manager.cc:1875] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:13.686822 21111 ts_tablet_manager.cc:1888] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:00:13.686892 21111 log.cc:1196] T 224b4919644d45dc942fb6654eb54c84 P b0da919a0055405e8ef4dfd545e3bd23: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/224b4919644d45dc942fb6654eb54c84
I20240502 14:00:13.687427 17279 catalog_manager.cc:4830] TS b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979): tablet 224b4919644d45dc942fb6654eb54c84 (table rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]) successfully deleted
I20240502 14:00:13.781678 17566 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:13.783680 17698 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:13.783965 18825 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:13.784090 21070 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:13.784428 21102 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:13.786140 21102 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:13.786614 17698 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
W20240502 14:00:13.788712 17501 consensus_peers.cc:485] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031): Couldn't send request to peer c4a5c8f4d3bb4689a4e38e1c743b1c0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7f702fa6854711b6afbebd76662177. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:13.788939 18825 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } } }
I20240502 14:00:13.788940 17294 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 7 to 8, NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:14.107224 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:14.215571 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:00:14.304399 21143 ts_tablet_manager.cc:902] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Initiating tablet copy from peer 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 14:00:14.304841 21143 tablet_copy_client.cc:308] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:42667
I20240502 14:00:14.307823 17588 tablet_copy_service.cc:140] P 7c2e9843dbba4851ad72d2d4a8b9886b: Received BeginTabletCopySession request for tablet 2d7f702fa6854711b6afbebd76662177 from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e ({username='slave'} at 127.16.217.8:43161)
I20240502 14:00:14.307932 17588 tablet_copy_service.cc:161] P 7c2e9843dbba4851ad72d2d4a8b9886b: Beginning new tablet copy session on tablet 2d7f702fa6854711b6afbebd76662177 from peer c4a5c8f4d3bb4689a4e38e1c743b1c0e at {username='slave'} at 127.16.217.8:43161: session id = c4a5c8f4d3bb4689a4e38e1c743b1c0e-2d7f702fa6854711b6afbebd76662177
I20240502 14:00:14.308696 17588 tablet_copy_source_session.cc:205] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:14.309470 21143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7f702fa6854711b6afbebd76662177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:14.311128 21143 tablet_copy_client.cc:783] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:14.311267 21143 tablet_copy_client.cc:647] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:14.312224 21147 tablet_copy_client.cc:1002] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:14.998380 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:15.238332 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:00:15.812527 21143 tablet_copy_client.cc:523] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:15.813831 21143 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap starting.
I20240502 14:00:15.830607 21143 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:15.830848 21143 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Bootstrap complete.
I20240502 14:00:15.831009 21143 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240502 14:00:15.831390 21143 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:15.831491 21143 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c4a5c8f4d3bb4689a4e38e1c743b1c0e, State: Initialized, Role: LEARNER
I20240502 14:00:15.831583 21143 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: NON_VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: true } }
I20240502 14:00:15.831879 21143 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:00:15.832491 17588 tablet_copy_service.cc:342] P 7c2e9843dbba4851ad72d2d4a8b9886b: Request end of tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-2d7f702fa6854711b6afbebd76662177 received from {username='slave'} at 127.16.217.8:43161
I20240502 14:00:15.832567 17588 tablet_copy_service.cc:433] P 7c2e9843dbba4851ad72d2d4a8b9886b: ending tablet copy session c4a5c8f4d3bb4689a4e38e1c743b1c0e-2d7f702fa6854711b6afbebd76662177 on tablet 2d7f702fa6854711b6afbebd76662177 with peer c4a5c8f4d3bb4689a4e38e1c743b1c0e
W20240502 14:00:15.869647 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:16.084008 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:00:16.146009 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:00:16.293661 19218 raft_consensus.cc:1203] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 14:00:16.380937 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a67fbce1caca46fa818f4e5d90a15305: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:16.480620 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:16.655319 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:00:16.753118 21179 raft_consensus.cc:1050] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b: attempting to promote NON_VOTER c4a5c8f4d3bb4689a4e38e1c743b1c0e to VOTER
I20240502 14:00:16.753508 21179 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:16.754750 19218 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 LEARNER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:16.754763 18825 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:16.755107 17697 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:16.755091 21222 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:16.755445 21222 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:16.755590 21224 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:16.757200 21224 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.757706 18825 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.758193 17697 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.758945 19218 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.760644 17294 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 reported cstate change: config changed from index 8 to 9, c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } attrs { replace: true } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.764027 17566 consensus_queue.cc:237] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } }
I20240502 14:00:16.764986 19218 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:16.765022 18825 raft_consensus.cc:1261] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:00:16.765300 21179 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:16.765448 21222 consensus_queue.cc:1035] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:16.766752 21216 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.767130 19218 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.767184 18825 raft_consensus.cc:2932] T 2d7f702fa6854711b6afbebd76662177 P 3305e392697b48a8ad63900628ce4c27 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } }
I20240502 14:00:16.769295 17290 catalog_manager.cc:5484] T 2d7f702fa6854711b6afbebd76662177 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 9 to 10, VOTER b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:16.770046 17280 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d7f702fa6854711b6afbebd76662177 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:16.771430 17679 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7f702fa6854711b6afbebd76662177 with delete_type TABLET_DATA_TOMBSTONED (TS b0da919a0055405e8ef4dfd545e3bd23 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60834
I20240502 14:00:16.771756 21233 tablet_replica.cc:330] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: stopping tablet replica
I20240502 14:00:16.771890 21233 raft_consensus.cc:2229] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:16.772055 21233 raft_consensus.cc:2258] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:16.772611 21233 ts_tablet_manager.cc:1875] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:16.774303 21233 ts_tablet_manager.cc:1888] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:00:16.774363 21233 log.cc:1196] T 2d7f702fa6854711b6afbebd76662177 P b0da919a0055405e8ef4dfd545e3bd23: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/2d7f702fa6854711b6afbebd76662177
I20240502 14:00:16.774784 17279 catalog_manager.cc:4830] TS b0da919a0055405e8ef4dfd545e3bd23 (127.16.217.3:42979): tablet 2d7f702fa6854711b6afbebd76662177 (table rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]) successfully deleted
I20240502 14:00:16.936354 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:16.938575 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.938853 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:16.943143 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.956616 18936 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:16.961982 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:16.964666 17809 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:16.993778 17829 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:16.995296 18956 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:16.995501 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 14:00:16.995671 21286 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:16.995852 21285 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:16.997437 21285 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:16.997947 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
W20240502 14:00:16.998248 17766 consensus_peers.cc:485] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 -> Peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123): Couldn't send request to peer d2dc7e8c2808471fb5a9a951c70d5c2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a67fbce1caca46fa818f4e5d90a15305. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:16.998250 18956 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } } }
I20240502 14:00:17.000432 17290 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 10 to 11, NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:17.007045 18956 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:17.008400 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:17.008612 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:17.008778 21137 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:17.009117 21137 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:17.010353 21137 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:17.010777 19218 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:17.011652 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:17.012974 17287 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 11 to 12, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:17.015650 18891 consensus_peers.cc:485] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b -> Peer 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713): Couldn't send request to peer 3305e392697b48a8ad63900628ce4c27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b05abb189494b6592787e2f17b4c33a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:17.359709 21289 consensus_queue.cc:579] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.095s)
I20240502 14:00:17.394481 21321 ts_tablet_manager.cc:902] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Initiating tablet copy from peer e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769)
I20240502 14:00:17.394914 21321 tablet_copy_client.cc:308] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:40769
I20240502 14:00:17.395344 17850 tablet_copy_service.cc:140] P e7f378d8d4c745d888ed99454393bb85: Received BeginTabletCopySession request for tablet a67fbce1caca46fa818f4e5d90a15305 from peer d2dc7e8c2808471fb5a9a951c70d5c2a ({username='slave'} at 127.16.217.7:59359)
I20240502 14:00:17.395443 17850 tablet_copy_service.cc:161] P e7f378d8d4c745d888ed99454393bb85: Beginning new tablet copy session on tablet a67fbce1caca46fa818f4e5d90a15305 from peer d2dc7e8c2808471fb5a9a951c70d5c2a at {username='slave'} at 127.16.217.7:59359: session id = d2dc7e8c2808471fb5a9a951c70d5c2a-a67fbce1caca46fa818f4e5d90a15305
I20240502 14:00:17.396148 17850 tablet_copy_source_session.cc:205] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:17.396906 21321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fbce1caca46fa818f4e5d90a15305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.398521 21321 tablet_copy_client.cc:783] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:17.398681 21321 tablet_copy_client.cc:647] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:17.399593 21325 tablet_copy_client.cc:1002] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Injecting latency into file download: 1500
I20240502 14:00:17.416254 21289 consensus_queue.cc:579] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.069s)
I20240502 14:00:17.525518 21336 ts_tablet_manager.cc:902] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Initiating tablet copy from peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157)
I20240502 14:00:17.525993 21336 tablet_copy_client.cc:308] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.6:40157
I20240502 14:00:17.529223 18976 tablet_copy_service.cc:140] P 73731152a18648aab3b1a720dabcc26b: Received BeginTabletCopySession request for tablet 5b05abb189494b6592787e2f17b4c33a from peer 3305e392697b48a8ad63900628ce4c27 ({username='slave'} at 127.16.217.5:50195)
I20240502 14:00:17.529330 18976 tablet_copy_service.cc:161] P 73731152a18648aab3b1a720dabcc26b: Beginning new tablet copy session on tablet 5b05abb189494b6592787e2f17b4c33a from peer 3305e392697b48a8ad63900628ce4c27 at {username='slave'} at 127.16.217.5:50195: session id = 3305e392697b48a8ad63900628ce4c27-5b05abb189494b6592787e2f17b4c33a
I20240502 14:00:17.530371 18976 tablet_copy_source_session.cc:205] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:17.531345 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b05abb189494b6592787e2f17b4c33a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:17.533077 21336 tablet_copy_client.cc:783] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:17.533222 21336 tablet_copy_client.cc:647] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:17.534293 21340 tablet_copy_client.cc:1002] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:17.565819 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:00:17.687875 21222 consensus_queue.cc:579] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.120s)
I20240502 14:00:17.771942 20996 consensus_queue.cc:579] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.241s)
W20240502 14:00:17.773221 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:17.959537 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:00:17.962076 21353 consensus_queue.cc:579] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.394s)
I20240502 14:00:18.225404 20996 consensus_queue.cc:579] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.543s)
W20240502 14:00:18.418084 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:18.693864 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:00:18.732474 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:00:18.786219 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:18.820672 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:00:18.899999 21321 tablet_copy_client.cc:523] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:18.901541 21321 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap starting.
I20240502 14:00:18.918637 21321 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:18.918852 21321 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Bootstrap complete.
I20240502 14:00:18.918977 21321 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20240502 14:00:18.919277 21321 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:18.919375 21321 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2dc7e8c2808471fb5a9a951c70d5c2a, State: Initialized, Role: LEARNER
I20240502 14:00:18.919452 21321 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: NON_VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: true } }
I20240502 14:00:18.919673 21321 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:18.920238 17850 tablet_copy_service.cc:342] P e7f378d8d4c745d888ed99454393bb85: Request end of tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-a67fbce1caca46fa818f4e5d90a15305 received from {username='slave'} at 127.16.217.7:59359
I20240502 14:00:18.920352 17850 tablet_copy_service.cc:433] P e7f378d8d4c745d888ed99454393bb85: ending tablet copy session d2dc7e8c2808471fb5a9a951c70d5c2a-a67fbce1caca46fa818f4e5d90a15305 on tablet a67fbce1caca46fa818f4e5d90a15305 with peer d2dc7e8c2808471fb5a9a951c70d5c2a
I20240502 14:00:19.034598 21336 tablet_copy_client.cc:523] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:19.035899 21336 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Bootstrap starting.
I20240502 14:00:19.056193 21336 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:19.056510 21336 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Bootstrap complete.
I20240502 14:00:19.056679 21336 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.005s
I20240502 14:00:19.057075 21336 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:19.057193 21336 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3305e392697b48a8ad63900628ce4c27, State: Initialized, Role: LEARNER
I20240502 14:00:19.057286 21336 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:19.057577 21336 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:19.058240 18976 tablet_copy_service.cc:342] P 73731152a18648aab3b1a720dabcc26b: Request end of tablet copy session 3305e392697b48a8ad63900628ce4c27-5b05abb189494b6592787e2f17b4c33a received from {username='slave'} at 127.16.217.5:50195
I20240502 14:00:19.058332 18976 tablet_copy_service.cc:433] P 73731152a18648aab3b1a720dabcc26b: ending tablet copy session 3305e392697b48a8ad63900628ce4c27-5b05abb189494b6592787e2f17b4c33a on tablet 5b05abb189494b6592787e2f17b4c33a with peer 3305e392697b48a8ad63900628ce4c27
I20240502 14:00:19.152387 19086 raft_consensus.cc:1203] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 14:00:19.172915 20996 consensus_queue.cc:579] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Leader has been unable to successfully communicate with peer b36b507eb389473b8438e907237a63f9 for more than 30 seconds (30.087s)
W20240502 14:00:19.174062 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:00:19.474918 18825 raft_consensus.cc:1203] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
W20240502 14:00:19.532461 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:00:19.533223 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4a240577417f4ae1812bac08ba71861b: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:19.738952 21455 raft_consensus.cc:1050] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: attempting to promote NON_VOTER d2dc7e8c2808471fb5a9a951c70d5c2a to VOTER
I20240502 14:00:19.739496 21455 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:19.740975 18956 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:19.741029 19086 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 LEARNER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:19.741490 21455 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:19.741542 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Refusing update from remote peer e7f378d8d4c745d888ed99454393bb85: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 14:00:19.741737 21454 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:19.741952 21431 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:19.743659 21454 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:19.744053 19218 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:19.745610 18956 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:19.746642 17288 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e reported cstate change: config changed from index 11 to 12, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7f378d8d4c745d888ed99454393bb85" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:19.747138 19086 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:19.775521 17829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305"
dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41022
I20240502 14:00:19.775653 17829 raft_consensus.cc:595] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:19.780448 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:00:19.863515 21481 raft_consensus.cc:1050] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b: attempting to promote NON_VOTER 3305e392697b48a8ad63900628ce4c27 to VOTER
I20240502 14:00:19.864051 21481 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:19.865434 18825 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:19.865499 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:19.865657 17566 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:19.865832 21480 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:19.866030 21483 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:19.866185 21481 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:19.868300 21480 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.868790 17566 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.869331 18825 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.870343 19218 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.871255 17294 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 12 to 13, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:19.874097 18956 consensus_queue.cc:237] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:19.875346 19218 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 14:00:19.875636 21480 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:19.875918 18825 raft_consensus.cc:1261] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 14:00:19.876292 21480 consensus_queue.cc:1035] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:19.877055 21481 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.878089 19218 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.878198 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b05abb189494b6592787e2f17b4c33a with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:19.878957 17288 catalog_manager.cc:5484] T 5b05abb189494b6592787e2f17b4c33a P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 13 to 14, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:19.879344 18825 raft_consensus.cc:2932] T 5b05abb189494b6592787e2f17b4c33a P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:19.881105 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b05abb189494b6592787e2f17b4c33a with delete_type TABLET_DATA_TOMBSTONED (TS 7c2e9843dbba4851ad72d2d4a8b9886b not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:54388
I20240502 14:00:19.881443 21487 tablet_replica.cc:330] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: stopping tablet replica
I20240502 14:00:19.881589 21487 raft_consensus.cc:2229] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:19.881839 21487 raft_consensus.cc:2258] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:19.882418 21487 ts_tablet_manager.cc:1875] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:19.883937 21487 ts_tablet_manager.cc:1888] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 14:00:19.883996 21487 log.cc:1196] T 5b05abb189494b6592787e2f17b4c33a P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/5b05abb189494b6592787e2f17b4c33a
I20240502 14:00:19.884464 17280 catalog_manager.cc:4830] TS 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667): tablet 5b05abb189494b6592787e2f17b4c33a (table rebalance_test_table_0 [id=5766b3d52639497aad4ac6afa2638f7c]) successfully deleted
I20240502 14:00:19.990175 17829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305"
dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41038
I20240502 14:00:19.990314 17829 raft_consensus.cc:595] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:19.997154 19086 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:19.998564 17566 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:00:19.998914 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:00:19.999248 21510 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:19.999505 21511 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:20.001192 21510 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:20.001657 17566 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
W20240502 14:00:20.002859 19023 consensus_peers.cc:485] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a -> Peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157): Couldn't send request to peer 73731152a18648aab3b1a720dabcc26b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda5fbedb8b54b56ab058434f4eb4395. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:20.003688 17287 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:20.004251 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:20.018633 17829 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305"
dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41064
I20240502 14:00:20.018756 17829 raft_consensus.cc:595] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:00:20.127322 21431 raft_consensus.cc:979] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: : Instructing follower 73731152a18648aab3b1a720dabcc26b to start an election
I20240502 14:00:20.127427 21431 raft_consensus.cc:1067] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Signalling peer 73731152a18648aab3b1a720dabcc26b to start an election
I20240502 14:00:20.127882 18956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305"
dest_uuid: "73731152a18648aab3b1a720dabcc26b"
from {username='slave'} at 127.16.217.4:47771
I20240502 14:00:20.128041 18956 raft_consensus.cc:482] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:20.128103 18956 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.128944 18956 raft_consensus.cc:504] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:20.129410 18956 leader_election.cc:288] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [CANDIDATE]: Term 2 election: Requested vote from peers e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769), c4a5c8f4d3bb4689a4e38e1c743b1c0e (127.16.217.8:36031), d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123)
I20240502 14:00:20.129675 17829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "e7f378d8d4c745d888ed99454393bb85"
I20240502 14:00:20.129796 17829 raft_consensus.cc:3032] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:20.129833 17829 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f378d8d4c745d888ed99454393bb85, State: Running, Role: LEADER
I20240502 14:00:20.129844 19218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e"
I20240502 14:00:20.129972 19218 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.129969 17829 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:20.130182 17829 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.130972 19218 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
I20240502 14:00:20.131271 17829 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
I20240502 14:00:20.131572 18891 leader_election.cc:302] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 73731152a18648aab3b1a720dabcc26b, c4a5c8f4d3bb4689a4e38e1c743b1c0e, e7f378d8d4c745d888ed99454393bb85; no voters:
I20240502 14:00:20.131726 21480 raft_consensus.cc:2781] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:20.131824 21480 raft_consensus.cc:686] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Becoming Leader. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Running, Role: LEADER
I20240502 14:00:20.131947 21480 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:20.133567 17287 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b reported cstate change: term changed from 1 to 2, leader changed from e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) to 73731152a18648aab3b1a720dabcc26b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:20.133724 19086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fbce1caca46fa818f4e5d90a15305" candidate_uuid: "73731152a18648aab3b1a720dabcc26b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a"
I20240502 14:00:20.133908 19086 raft_consensus.cc:3037] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:20.134817 19086 raft_consensus.cc:2454] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73731152a18648aab3b1a720dabcc26b in term 2.
W20240502 14:00:20.211524 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:00:20.518155 21551 ts_tablet_manager.cc:902] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Initiating tablet copy from peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123)
I20240502 14:00:20.518638 21551 tablet_copy_client.cc:308] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.7:44123
I20240502 14:00:20.519196 19107 tablet_copy_service.cc:140] P d2dc7e8c2808471fb5a9a951c70d5c2a: Received BeginTabletCopySession request for tablet dda5fbedb8b54b56ab058434f4eb4395 from peer 73731152a18648aab3b1a720dabcc26b ({username='slave'} at 127.16.217.6:57775)
I20240502 14:00:20.519332 19107 tablet_copy_service.cc:161] P d2dc7e8c2808471fb5a9a951c70d5c2a: Beginning new tablet copy session on tablet dda5fbedb8b54b56ab058434f4eb4395 from peer 73731152a18648aab3b1a720dabcc26b at {username='slave'} at 127.16.217.6:57775: session id = 73731152a18648aab3b1a720dabcc26b-dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:20.520417 19107 tablet_copy_source_session.cc:205] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:20.521436 21551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:20.523121 21551 tablet_copy_client.cc:783] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:20.523262 21551 tablet_copy_client.cc:647] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:20.524358 19218 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:20.524528 21555 tablet_copy_client.cc:1002] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Injecting latency into file download: 1500
I20240502 14:00:20.524751 21480 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:20.622108 19086 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:20.622536 21480 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:20.624892 17829 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:20.625392 21558 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:20.627581 18956 consensus_queue.cc:237] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } }
I20240502 14:00:20.628707 19086 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 14:00:20.628726 19217 raft_consensus.cc:1261] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Refusing update from remote peer 73731152a18648aab3b1a720dabcc26b: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 14:00:20.629000 21558 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:20.629101 21480 consensus_queue.cc:1035] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:00:20.630429 21483 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:20.630853 19217 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P c4a5c8f4d3bb4689a4e38e1c743b1c0e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:20.632225 19086 raft_consensus.cc:2932] T a67fbce1caca46fa818f4e5d90a15305 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } }
I20240502 14:00:20.632366 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a67fbce1caca46fa818f4e5d90a15305 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:20.632927 17287 catalog_manager.cc:5484] T a67fbce1caca46fa818f4e5d90a15305 P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 12 to 14, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New cstate: current_term: 2 leader_uuid: "73731152a18648aab3b1a720dabcc26b" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c4a5c8f4d3bb4689a4e38e1c743b1c0e" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 36031 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:20.635073 17809 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fbce1caca46fa818f4e5d90a15305 with delete_type TABLET_DATA_TOMBSTONED (TS e7f378d8d4c745d888ed99454393bb85 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:44318
I20240502 14:00:20.635385 21562 tablet_replica.cc:330] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: stopping tablet replica
I20240502 14:00:20.635556 21562 raft_consensus.cc:2229] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:20.635715 21562 raft_consensus.cc:2258] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:20.636368 21562 ts_tablet_manager.cc:1875] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:20.638214 21562 ts_tablet_manager.cc:1888] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 14:00:20.638270 21562 log.cc:1196] T a67fbce1caca46fa818f4e5d90a15305 P e7f378d8d4c745d888ed99454393bb85: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/a67fbce1caca46fa818f4e5d90a15305
I20240502 14:00:20.638681 17281 catalog_manager.cc:4830] TS e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769): tablet a67fbce1caca46fa818f4e5d90a15305 (table rebalance_test_table_4 [id=1a50cb441a2148e9bf3a4dd5441f05e7]) successfully deleted
W20240502 14:00:20.970506 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:21.248247 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:21.431205 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:21.555230 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:00:21.748934 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:00:22.024820 21551 tablet_copy_client.cc:523] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:22.026203 21551 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Bootstrap starting.
I20240502 14:00:22.045130 21551 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:22.045432 21551 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Bootstrap complete.
I20240502 14:00:22.045594 21551 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240502 14:00:22.045969 21551 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:22.046069 21551 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Initialized, Role: LEARNER
I20240502 14:00:22.046165 21551 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:22.046423 21551 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:22.047031 19107 tablet_copy_service.cc:342] P d2dc7e8c2808471fb5a9a951c70d5c2a: Request end of tablet copy session 73731152a18648aab3b1a720dabcc26b-dda5fbedb8b54b56ab058434f4eb4395 received from {username='slave'} at 127.16.217.6:57775
I20240502 14:00:22.047116 19107 tablet_copy_service.cc:433] P d2dc7e8c2808471fb5a9a951c70d5c2a: ending tablet copy session 73731152a18648aab3b1a720dabcc26b-dda5fbedb8b54b56ab058434f4eb4395 on tablet dda5fbedb8b54b56ab058434f4eb4395 with peer 73731152a18648aab3b1a720dabcc26b
W20240502 14:00:22.291049 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:00:22.310145 18956 raft_consensus.cc:1203] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20240502 14:00:22.766734 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:00:22.805369 21584 raft_consensus.cc:1050] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: attempting to promote NON_VOTER 73731152a18648aab3b1a720dabcc26b to VOTER
I20240502 14:00:22.805738 21584 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:22.807004 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:00:22.807113 17566 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:00:22.807463 21635 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:22.807749 21637 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:22.807933 18955 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 LEARNER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:00:22.808409 21635 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:22.810668 18956 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.810937 17566 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.810519 21637 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.812919 17287 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b reported cstate change: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.813707 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } attrs { replace: true } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.816634 19086 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:22.817523 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:00:22.817585 18956 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:00:22.817909 21638 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:22.818081 21637 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:00:22.819540 21635 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.819973 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.820003 18956 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:22.822209 17294 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 9 to 10, VOTER 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:22.822335 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dda5fbedb8b54b56ab058434f4eb4395 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:22.824154 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 with delete_type TABLET_DATA_TOMBSTONED (TS 7c2e9843dbba4851ad72d2d4a8b9886b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54388
I20240502 14:00:22.824905 21647 tablet_replica.cc:330] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: stopping tablet replica
I20240502 14:00:22.825084 21647 raft_consensus.cc:2229] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:22.825345 21647 raft_consensus.cc:2258] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:22.826020 21647 ts_tablet_manager.cc:1875] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:22.827582 21647 ts_tablet_manager.cc:1888] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:00:22.827649 21647 log.cc:1196] T dda5fbedb8b54b56ab058434f4eb4395 P 7c2e9843dbba4851ad72d2d4a8b9886b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:22.828212 17280 catalog_manager.cc:4830] TS 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667): tablet dda5fbedb8b54b56ab058434f4eb4395 (table rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]) successfully deleted
I20240502 14:00:22.924033 17566 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:22.925678 19086 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:22.925884 17829 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:00:22.926298 21661 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:22.926481 21660 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:00:22.928191 21661 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
I20240502 14:00:22.928840 17829 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
W20240502 14:00:22.929301 17504 consensus_peers.cc:485] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer 73731152a18648aab3b1a720dabcc26b (127.16.217.6:40157): Couldn't send request to peer 73731152a18648aab3b1a720dabcc26b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 224b4919644d45dc942fb6654eb54c84. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:22.931214 17290 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 6 to 7, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:00:22.931793 19086 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 73731152a18648aab3b1a720dabcc26b (127.16.217.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } } }
W20240502 14:00:23.032420 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:00:23.330065 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5b05abb189494b6592787e2f17b4c33a: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
I20240502 14:00:23.333072 21678 ts_tablet_manager.cc:902] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Initiating tablet copy from peer 7c2e9843dbba4851ad72d2d4a8b9886b (127.16.217.2:42667)
I20240502 14:00:23.333474 21678 tablet_copy_client.cc:308] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:42667
I20240502 14:00:23.333876 17588 tablet_copy_service.cc:140] P 7c2e9843dbba4851ad72d2d4a8b9886b: Received BeginTabletCopySession request for tablet 224b4919644d45dc942fb6654eb54c84 from peer 73731152a18648aab3b1a720dabcc26b ({username='slave'} at 127.16.217.6:51167)
I20240502 14:00:23.333964 17588 tablet_copy_service.cc:161] P 7c2e9843dbba4851ad72d2d4a8b9886b: Beginning new tablet copy session on tablet 224b4919644d45dc942fb6654eb54c84 from peer 73731152a18648aab3b1a720dabcc26b at {username='slave'} at 127.16.217.6:51167: session id = 73731152a18648aab3b1a720dabcc26b-224b4919644d45dc942fb6654eb54c84
I20240502 14:00:23.334614 17588 tablet_copy_source_session.cc:205] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:23.335326 21678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224b4919644d45dc942fb6654eb54c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:23.336910 21678 tablet_copy_client.cc:783] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:23.337096 21678 tablet_copy_client.cc:647] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:23.337881 21682 tablet_copy_client.cc:1002] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:23.467700 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:23.823586 17502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:00:23.824126 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:23.929056 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:24.071142 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:24.178681 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:00:24.689374 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:00:24.838157 21678 tablet_copy_client.cc:523] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:24.839337 21678 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Bootstrap starting.
I20240502 14:00:24.856686 21678 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:24.856881 21678 tablet_bootstrap.cc:492] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Bootstrap complete.
I20240502 14:00:24.856997 21678 ts_tablet_manager.cc:1367] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.005s
I20240502 14:00:24.857286 21678 raft_consensus.cc:348] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:24.857376 21678 raft_consensus.cc:729] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73731152a18648aab3b1a720dabcc26b, State: Initialized, Role: LEARNER
I20240502 14:00:24.857442 21678 consensus_queue.cc:260] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:24.857645 21678 ts_tablet_manager.cc:1398] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:24.858001 17588 tablet_copy_service.cc:342] P 7c2e9843dbba4851ad72d2d4a8b9886b: Request end of tablet copy session 73731152a18648aab3b1a720dabcc26b-224b4919644d45dc942fb6654eb54c84 received from {username='slave'} at 127.16.217.6:51167
I20240502 14:00:24.858064 17588 tablet_copy_service.cc:433] P 7c2e9843dbba4851ad72d2d4a8b9886b: ending tablet copy session 73731152a18648aab3b1a720dabcc26b-224b4919644d45dc942fb6654eb54c84 on tablet 224b4919644d45dc942fb6654eb54c84 with peer 73731152a18648aab3b1a720dabcc26b
I20240502 14:00:24.924883 18956 raft_consensus.cc:1203] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 14:00:25.116428 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:00:25.320061 21740 raft_consensus.cc:1050] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b: attempting to promote NON_VOTER 73731152a18648aab3b1a720dabcc26b to VOTER
I20240502 14:00:25.320497 21740 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:25.321849 18956 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 LEARNER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:25.321849 19086 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:25.321887 17829 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:00:25.322249 21740 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:25.322414 21743 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:25.322557 21742 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:00:25.324110 21742 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.324587 17829 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.324600 19086 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.326593 17290 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 reported cstate change: config changed from index 7 to 8, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.326988 18956 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 73731152a18648aab3b1a720dabcc26b (127.16.217.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.330253 17566 consensus_queue.cc:237] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } }
I20240502 14:00:25.331084 18956 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:25.331148 19086 raft_consensus.cc:1261] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Refusing update from remote peer 7c2e9843dbba4851ad72d2d4a8b9886b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:00:25.331391 21717 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:25.331532 21740 consensus_queue.cc:1035] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:00:25.332901 21740 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.334281 17280 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 224b4919644d45dc942fb6654eb54c84 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:25.333844 19086 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.334619 18956 raft_consensus.cc:2932] T 224b4919644d45dc942fb6654eb54c84 P 73731152a18648aab3b1a720dabcc26b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } }
I20240502 14:00:25.335969 17288 catalog_manager.cc:5484] T 224b4919644d45dc942fb6654eb54c84 P 7c2e9843dbba4851ad72d2d4a8b9886b reported cstate change: config changed from index 8 to 9, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:25.337682 17809 tablet_service.cc:1514] Processing DeleteTablet for tablet 224b4919644d45dc942fb6654eb54c84 with delete_type TABLET_DATA_TOMBSTONED (TS e7f378d8d4c745d888ed99454393bb85 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:44318
I20240502 14:00:25.338068 21751 tablet_replica.cc:330] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: stopping tablet replica
I20240502 14:00:25.338201 21751 raft_consensus.cc:2229] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:25.338364 21751 raft_consensus.cc:2258] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:25.338891 21751 ts_tablet_manager.cc:1875] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:25.340595 21751 ts_tablet_manager.cc:1888] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:00:25.340657 21751 log.cc:1196] T 224b4919644d45dc942fb6654eb54c84 P e7f378d8d4c745d888ed99454393bb85: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/224b4919644d45dc942fb6654eb54c84
I20240502 14:00:25.341117 17281 catalog_manager.cc:4830] TS e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769): tablet 224b4919644d45dc942fb6654eb54c84 (table rebalance_test_table_1 [id=009ecb6e9a1d4695a9a570366fd050b6]) successfully deleted
I20240502 14:00:25.455418 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:25.455549 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:25.458513 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:25.459502 18936 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:25.465528 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:25.472672 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:25.487476 17809 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:25.512415 19086 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:25.513753 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:00:25.513797 18956 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:00:25.514111 21803 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:25.514403 21803 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:00:25.515578 21803 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:25.515972 18956 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:25.516728 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } } }
I20240502 14:00:25.518420 17291 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 10 to 11, NON_VOTER 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:00:25.518776 19022 consensus_peers.cc:485] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a -> Peer 3305e392697b48a8ad63900628ce4c27 (127.16.217.5:35713): Couldn't send request to peer 3305e392697b48a8ad63900628ce4c27. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dda5fbedb8b54b56ab058434f4eb4395. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:00:25.570027 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:00:26.011216 21822 ts_tablet_manager.cc:902] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Initiating tablet copy from peer d2dc7e8c2808471fb5a9a951c70d5c2a (127.16.217.7:44123)
I20240502 14:00:26.011693 21822 tablet_copy_client.cc:308] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.7:44123
I20240502 14:00:26.014307 19107 tablet_copy_service.cc:140] P d2dc7e8c2808471fb5a9a951c70d5c2a: Received BeginTabletCopySession request for tablet dda5fbedb8b54b56ab058434f4eb4395 from peer 3305e392697b48a8ad63900628ce4c27 ({username='slave'} at 127.16.217.5:37357)
I20240502 14:00:26.014389 19107 tablet_copy_service.cc:161] P d2dc7e8c2808471fb5a9a951c70d5c2a: Beginning new tablet copy session on tablet dda5fbedb8b54b56ab058434f4eb4395 from peer 3305e392697b48a8ad63900628ce4c27 at {username='slave'} at 127.16.217.5:37357: session id = 3305e392697b48a8ad63900628ce4c27-dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:26.015033 19107 tablet_copy_source_session.cc:205] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:26.015729 21822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dda5fbedb8b54b56ab058434f4eb4395. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:26.017302 21822 tablet_copy_client.cc:783] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:26.017429 21822 tablet_copy_client.cc:647] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:26.018251 21826 tablet_copy_client.cc:1002] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Injecting latency into file download: 1500
W20240502 14:00:26.264626 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:26.277953 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:26.370735 17764 consensus_peers.cc:485] T 588123f560c04bc4b9a3d3c5c4251135 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:26.481133 17280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 14:00:26.481740 17280 catalog_manager.cc:4573] TS b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 30459a30f0de47ba86791dd018538bce: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111)
W20240502 14:00:26.556532 17764 consensus_peers.cc:485] T 6c78e509dd8e4e5e8884fa539c9c3f6f P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:00:26.824168 17502 consensus_peers.cc:485] T 37d38fe2e1de4c0284982425d25829a7 P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:27.467396 17764 consensus_peers.cc:485] T be290fd1e9464cc9ba4024b6ec760683 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:00:27.518505 21822 tablet_copy_client.cc:523] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:27.519645 21822 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Bootstrap starting.
I20240502 14:00:27.536922 21822 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:27.537138 21822 tablet_bootstrap.cc:492] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Bootstrap complete.
I20240502 14:00:27.537276 21822 ts_tablet_manager.cc:1367] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240502 14:00:27.537597 21822 raft_consensus.cc:348] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:27.537688 21822 raft_consensus.cc:729] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3305e392697b48a8ad63900628ce4c27, State: Initialized, Role: LEARNER
I20240502 14:00:27.537770 21822 consensus_queue.cc:260] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: NON_VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: true } }
I20240502 14:00:27.538038 21822 ts_tablet_manager.cc:1398] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:27.538462 19107 tablet_copy_service.cc:342] P d2dc7e8c2808471fb5a9a951c70d5c2a: Request end of tablet copy session 3305e392697b48a8ad63900628ce4c27-dda5fbedb8b54b56ab058434f4eb4395 received from {username='slave'} at 127.16.217.5:37357
I20240502 14:00:27.538532 19107 tablet_copy_service.cc:433] P d2dc7e8c2808471fb5a9a951c70d5c2a: ending tablet copy session 3305e392697b48a8ad63900628ce4c27-dda5fbedb8b54b56ab058434f4eb4395 on tablet dda5fbedb8b54b56ab058434f4eb4395 with peer 3305e392697b48a8ad63900628ce4c27
I20240502 14:00:27.647333 18825 raft_consensus.cc:1203] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240502 14:00:27.813781 17764 consensus_peers.cc:485] T 126c4156e9da4208bdba9c3e778f4614 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:00:28.037417 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.1:35657: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240502 14:00:28.120692 21886 raft_consensus.cc:1050] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a: attempting to promote NON_VOTER 3305e392697b48a8ad63900628ce4c27 to VOTER
I20240502 14:00:28.121091 21886 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:28.122334 18825 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 LEARNER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:28.122354 18956 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:28.122704 21888 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:28.122783 17829 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 14:00:28.122922 21858 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:28.123102 21889 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:00:28.125280 21889 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.125664 17829 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.125720 18825 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.127084 18956 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.127835 17291 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 11 to 12, 3305e392697b48a8ad63900628ce4c27 (127.16.217.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:28.131156 19086 consensus_queue.cc:237] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } }
I20240502 14:00:28.132191 18825 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:28.132246 18956 raft_consensus.cc:1261] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Refusing update from remote peer d2dc7e8c2808471fb5a9a951c70d5c2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 14:00:28.132532 21858 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:28.132669 21886 consensus_queue.cc:1035] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:00:28.133975 21888 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.134361 18825 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 3305e392697b48a8ad63900628ce4c27 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.134388 18956 raft_consensus.cc:2932] T dda5fbedb8b54b56ab058434f4eb4395 P 73731152a18648aab3b1a720dabcc26b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } } }
I20240502 14:00:28.136179 17294 catalog_manager.cc:5484] T dda5fbedb8b54b56ab058434f4eb4395 P d2dc7e8c2808471fb5a9a951c70d5c2a reported cstate change: config changed from index 12 to 13, VOTER e7f378d8d4c745d888ed99454393bb85 (127.16.217.4) evicted. New cstate: current_term: 2 leader_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d2dc7e8c2808471fb5a9a951c70d5c2a" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 44123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3305e392697b48a8ad63900628ce4c27" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 35713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:00:28.136436 17282 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dda5fbedb8b54b56ab058434f4eb4395 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:00:28.138252 17809 tablet_service.cc:1514] Processing DeleteTablet for tablet dda5fbedb8b54b56ab058434f4eb4395 with delete_type TABLET_DATA_TOMBSTONED (TS e7f378d8d4c745d888ed99454393bb85 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:44318
I20240502 14:00:28.138516 21897 tablet_replica.cc:330] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: stopping tablet replica
I20240502 14:00:28.138634 21897 raft_consensus.cc:2229] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:28.138890 21897 raft_consensus.cc:2258] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:28.139482 21897 ts_tablet_manager.cc:1875] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:28.141014 21897 ts_tablet_manager.cc:1888] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 14:00:28.141072 21897 log.cc:1196] T dda5fbedb8b54b56ab058434f4eb4395 P e7f378d8d4c745d888ed99454393bb85: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/dda5fbedb8b54b56ab058434f4eb4395
I20240502 14:00:28.141466 17281 catalog_manager.cc:4830] TS e7f378d8d4c745d888ed99454393bb85 (127.16.217.4:40769): tablet dda5fbedb8b54b56ab058434f4eb4395 (table rebalance_test_table_3 [id=1b55906aa33647cf9f83c8f2230dfb35]) successfully deleted
I20240502 14:00:28.246480 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.246749 18936 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:28.248296 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.248618 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.259714 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.260792 17809 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:28.266649 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:28.334247 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.337956 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.338307 18936 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:28.338539 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:28.340409 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.343104 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.356432 17809 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:28.389107 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:35657
--local_ip_for_outbound_sockets=127.16.217.1
--tserver_master_addrs=127.16.217.62:46605
--webserver_port=36265
--webserver_interface=127.16.217.1
--builtin_ntp_servers=127.16.217.20:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 14:00:28.487329 21975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:28.487475 21975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:28.487512 21975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:28.490157 21975 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:00:28.490206 21975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:28.490259 21975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20240502 14:00:28.494201 21975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:35657
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=36265
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.217.62:46605
--never_fsync=true
--heap_profile_path=/tmp/kudu.21975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:28.494935 21975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:28.495952 21975 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:28.504091 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:28.504494 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:28.505227 21975 server_base.cc:1034] running on GCE node
W20240502 14:00:28.505275 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:28.505558 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:28.505980 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:28.507112 21975 hybrid_clock.cc:648] HybridClock initialized: now 1714658428507104 us; error 25 us; skew 500 ppm
I20240502 14:00:28.508729 21975 webserver.cc:460] Webserver started at http://127.16.217.1:36265/ using document root <none> and password file <none>
I20240502 14:00:28.509210 21975 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:28.509274 21975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:28.512055 21975 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:28.514317 21989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:28.515180 21975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:28.515269 21975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b36b507eb389473b8438e907237a63f9"
format_stamp: "Formatted at 2024-05-02 13:59:38 on dist-test-slave-7msf"
I20240502 14:00:28.515561 21975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:28.532856 21975 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:28.533399 21975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:28.533529 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:28.533991 21975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:28.535091 21996 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240502 14:00:28.543933 21975 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 14:00:28.543996 21975 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20240502 14:00:28.544064 21975 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 14:00:28.546041 21996 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.549345 21975 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 14:00:28.549388 21975 ts_tablet_manager.cc:569] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:28.558918 21975 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:35657
I20240502 14:00:28.558957 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:35657 every 8 connection(s)
I20240502 14:00:28.559525 21996 log.cc:824] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:28.560400 21975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:00:28.564498 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 21975
I20240502 14:00:28.569936 22105 heartbeater.cc:347] Connected to a master server at 127.16.217.62:46605
I20240502 14:00:28.570107 22105 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:28.570624 22105 heartbeater.cc:511] Master 127.16.217.62:46605 requested a full tablet report, sending...
I20240502 14:00:28.572377 17291 ts_manager.cc:194] Re-registered known tserver with Master: b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657)
I20240502 14:00:28.574617 17291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:34387
I20240502 14:00:28.589828 21996 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.590225 21996 tablet_bootstrap.cc:492] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.591428 21996 ts_tablet_manager.cc:1367] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.010s
I20240502 14:00:28.593585 21996 raft_consensus.cc:348] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 14:00:28.593933 21996 raft_consensus.cc:729] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.594341 21996 consensus_queue.cc:260] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 14:00:28.594997 22105 heartbeater.cc:503] Master 127.16.217.62:46605 was elected leader, sending a full tablet report...
I20240502 14:00:28.595088 21996 ts_tablet_manager.cc:1398] T 4a240577417f4ae1812bac08ba71861b P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 14:00:28.595309 21996 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.624032 21996 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.624372 21996 tablet_bootstrap.cc:492] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.625273 21996 ts_tablet_manager.cc:1367] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.006s
I20240502 14:00:28.625677 21996 raft_consensus.cc:348] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.625779 21996 raft_consensus.cc:729] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.625893 21996 consensus_queue.cc:260] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.626102 21996 ts_tablet_manager.cc:1398] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.626317 21996 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
W20240502 14:00:28.630601 17502 consensus_peers.cc:485] T d0d96979d0484944b737aa3e7452947d P 7c2e9843dbba4851ad72d2d4a8b9886b -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 81: this message will repeat every 5th retry.
I20240502 14:00:28.652969 21996 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.653276 21996 tablet_bootstrap.cc:492] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.654263 21996 ts_tablet_manager.cc:1367] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20240502 14:00:28.654651 21996 raft_consensus.cc:348] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.654760 21996 raft_consensus.cc:729] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.654875 21996 consensus_queue.cc:260] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.655124 21996 ts_tablet_manager.cc:1398] T 6c78e509dd8e4e5e8884fa539c9c3f6f P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.655249 21996 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.688427 21996 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.688786 21996 tablet_bootstrap.cc:492] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.689889 21996 ts_tablet_manager.cc:1367] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.003s
I20240502 14:00:28.690296 21996 raft_consensus.cc:348] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.690413 21996 raft_consensus.cc:729] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.690543 21996 consensus_queue.cc:260] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.690810 21996 ts_tablet_manager.cc:1398] T d0d96979d0484944b737aa3e7452947d P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.690949 21996 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
W20240502 14:00:28.697961 17764 consensus_peers.cc:485] T f8f878ad1d9240b79a9b8dbfc50f62d2 P e7f378d8d4c745d888ed99454393bb85 -> Peer b36b507eb389473b8438e907237a63f9 (127.16.217.1:35657): Couldn't send request to peer b36b507eb389473b8438e907237a63f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240502 14:00:28.718734 21996 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.719058 21996 tablet_bootstrap.cc:492] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.720064 21996 ts_tablet_manager.cc:1367] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.004s
I20240502 14:00:28.720409 21996 raft_consensus.cc:348] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.720494 21996 raft_consensus.cc:729] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.720624 21996 consensus_queue.cc:260] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.720841 21996 ts_tablet_manager.cc:1398] T 30459a30f0de47ba86791dd018538bce P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:00:28.721220 21996 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.741860 21996 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.742204 21996 tablet_bootstrap.cc:492] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.742962 21996 ts_tablet_manager.cc:1367] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20240502 14:00:28.743294 21996 raft_consensus.cc:348] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.743373 21996 raft_consensus.cc:729] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.743459 21996 consensus_queue.cc:260] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.743671 21996 ts_tablet_manager.cc:1398] T 126c4156e9da4208bdba9c3e778f4614 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.743850 21996 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.768580 21996 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.768950 21996 tablet_bootstrap.cc:492] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.769805 21996 ts_tablet_manager.cc:1367] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.003s
I20240502 14:00:28.770113 21996 raft_consensus.cc:348] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.770220 21996 raft_consensus.cc:729] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.770349 21996 consensus_queue.cc:260] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.770888 21996 ts_tablet_manager.cc:1398] T a67fbce1caca46fa818f4e5d90a15305 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:28.771036 21996 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.786963 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.787613 18936 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:28.797066 21996 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.797341 21996 tablet_bootstrap.cc:492] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.797413 22038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.798310 21996 ts_tablet_manager.cc:1367] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20240502 14:00:28.798625 21996 raft_consensus.cc:348] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.798707 21996 raft_consensus.cc:729] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.798802 21996 consensus_queue.cc:260] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } }
I20240502 14:00:28.799001 21996 ts_tablet_manager.cc:1398] T be290fd1e9464cc9ba4024b6ec760683 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.799124 21996 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.809083 17809 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:28.810874 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.812240 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:28.831593 21996 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.831923 21996 tablet_bootstrap.cc:492] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.832917 21996 ts_tablet_manager.cc:1367] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.009s
I20240502 14:00:28.833201 21996 raft_consensus.cc:348] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 14:00:28.833281 21996 raft_consensus.cc:729] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.833400 21996 consensus_queue.cc:260] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } } peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } }
I20240502 14:00:28.833608 21996 ts_tablet_manager.cc:1398] T 2d7f702fa6854711b6afbebd76662177 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.833738 21996 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.839747 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.839779 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.864360 21996 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.864688 21996 tablet_bootstrap.cc:492] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.865552 21996 ts_tablet_manager.cc:1367] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.005s
I20240502 14:00:28.865892 21996 raft_consensus.cc:348] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.866005 21996 raft_consensus.cc:729] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.866150 21996 consensus_queue.cc:260] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0da919a0055405e8ef4dfd545e3bd23" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 42979 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.866392 21996 ts_tablet_manager.cc:1398] T f8f878ad1d9240b79a9b8dbfc50f62d2 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.866552 21996 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.867724 22050 raft_consensus.cc:3037] T 588123f560c04bc4b9a3d3c5c4251135 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:28.889955 21996 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.890216 21996 tablet_bootstrap.cc:492] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.890925 21996 ts_tablet_manager.cc:1367] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20240502 14:00:28.891299 21996 raft_consensus.cc:348] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:28.891444 21996 raft_consensus.cc:729] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.891574 21996 consensus_queue.cc:260] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } attrs { replace: true } } peers { permanent_uuid: "73731152a18648aab3b1a720dabcc26b" member_type: NON_VOTER last_known_addr { host: "127.16.217.6" port: 40157 } attrs { promote: true } }
I20240502 14:00:28.891829 21996 ts_tablet_manager.cc:1398] T 5b05abb189494b6592787e2f17b4c33a P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.892019 21996 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Bootstrap starting.
I20240502 14:00:28.919530 21996 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:28.919868 21996 tablet_bootstrap.cc:492] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Bootstrap complete.
I20240502 14:00:28.920770 21996 ts_tablet_manager.cc:1367] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.002s
I20240502 14:00:28.921079 21996 raft_consensus.cc:348] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.921155 21996 raft_consensus.cc:729] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36b507eb389473b8438e907237a63f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:28.921270 21996 consensus_queue.cc:260] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c2e9843dbba4851ad72d2d4a8b9886b" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 42667 } } peers { permanent_uuid: "e7f378d8d4c745d888ed99454393bb85" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 40769 } } peers { permanent_uuid: "b36b507eb389473b8438e907237a63f9" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35657 } }
I20240502 14:00:28.921511 21996 ts_tablet_manager.cc:1398] T 37d38fe2e1de4c0284982425d25829a7 P b36b507eb389473b8438e907237a63f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:28.935055 17679 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.935842 18805 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.937517 22038 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.939016 19198 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:28.940474 17809 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:28.953711 18936 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:28.953701 17546 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:28.960882 19067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:28.992164 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 21975
I20240502 14:00:28.997840 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17486
I20240502 14:00:29.003883 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17617
I20240502 14:00:29.009816 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17748
I20240502 14:00:29.015589 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 18743
I20240502 14:00:29.020902 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 18874
I20240502 14:00:29.026104 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 19005
I20240502 14:00:29.031452 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 19136
I20240502 14:00:29.037421 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 17264
2024-05-02T14:00:29Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (51185 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (51185 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 14:00:29.070652 17252 test_util.cc:268] Using random seed: 110325050
I20240502 14:00:29.071296 17252 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:00:29.071336 17252 ts_itest-base.cc:113] --------------
I20240502 14:00:29.071346 17252 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:00:29.071357 17252 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:00:29.071374 17252 ts_itest-base.cc:116] --------------
2024-05-02T14:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:00:29Z Disabled control of system clock
I20240502 14:00:29.078086 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43279
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:43279
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:00:29.170897 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.171123 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.171195 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.173873 22209 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.173944 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:00:29.173975 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.174000 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:00:29.174022 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:00:29.176947 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:43279
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.177526 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:29.178315 22209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.183810 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.184301 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.184741 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.185425 22209 server_base.cc:1034] running on GCE node
I20240502 14:00:29.185778 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:29.186426 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:29.187577 22209 hybrid_clock.cc:648] HybridClock initialized: now 1714658429187567 us; error 29 us; skew 500 ppm
I20240502 14:00:29.193452 22209 webserver.cc:460] Webserver started at http://127.16.217.62:36219/ using document root <none> and password file <none>
I20240502 14:00:29.193878 22209 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.193928 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.194061 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.195291 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2a02118149134e488b5832458930c0d2"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.195672 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2a02118149134e488b5832458930c0d2"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.198220 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 14:00:29.199955 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.200716 22209 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:29.200810 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2a02118149134e488b5832458930c0d2"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.200876 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.219507 22209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.219949 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:29.220063 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.225260 22209 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:43279
I20240502 14:00:29.225266 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:43279 every 8 connection(s)
I20240502 14:00:29.225934 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:00:29.227985 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:29.231901 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2: Bootstrap starting.
I20240502 14:00:29.232136 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22209
I20240502 14:00:29.232266 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:00:29.234138 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:29.234853 22276 log.cc:824] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:29.236313 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2: No bootstrap required, opened a new log
I20240502 14:00:29.238814 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } }
I20240502 14:00:29.238950 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:29.238981 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a02118149134e488b5832458930c0d2, State: Initialized, Role: FOLLOWER
I20240502 14:00:29.239336 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } }
I20240502 14:00:29.239462 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:29.239513 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:29.239602 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:29.240149 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } }
I20240502 14:00:29.240419 22276 leader_election.cc:302] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a02118149134e488b5832458930c0d2; no voters:
I20240502 14:00:29.240604 22276 leader_election.cc:288] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:29.240680 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:29.241189 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a02118149134e488b5832458930c0d2, State: Running, Role: LEADER
I20240502 14:00:29.241418 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:29.241429 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } }
I20240502 14:00:29.242566 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a02118149134e488b5832458930c0d2. Latest consensus state: current_term: 1 leader_uuid: "2a02118149134e488b5832458930c0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } } }
I20240502 14:00:29.242672 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:29.242935 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a02118149134e488b5832458930c0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a02118149134e488b5832458930c0d2" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 43279 } } }
I20240502 14:00:29.243018 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2 [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:29.243220 22292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:29.245616 22292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:29.249202 22292 catalog_manager.cc:1347] Generated new cluster ID: 9f82a27d8f494b9dabcbbdb61deabc19
I20240502 14:00:29.249250 22292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:29.266635 22292 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:29.267588 22292 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:29.280807 22292 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2a02118149134e488b5832458930c0d2: Generated new TSK 0
I20240502 14:00:29.281316 22292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:29.295195 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:29.385442 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.385603 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.385641 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.388104 22300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.388151 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.388211 22300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20240502 14:00:29.391157 22300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--never_fsync=true
--heap_profile_path=/tmp/kudu.22300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.391767 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:29.392625 22300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.398878 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.399390 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.399902 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.400575 22300 server_base.cc:1034] running on GCE node
I20240502 14:00:29.400852 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:29.401295 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:29.402429 22300 hybrid_clock.cc:648] HybridClock initialized: now 1714658429402410 us; error 31 us; skew 500 ppm
I20240502 14:00:29.403936 22300 webserver.cc:460] Webserver started at http://127.16.217.1:37563/ using document root <none> and password file <none>
I20240502 14:00:29.404412 22300 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.404461 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.404613 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.405864 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a8ada5937bef43679cc3429ce96d2859"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.406248 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a8ada5937bef43679cc3429ce96d2859"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.408895 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:00:29.410640 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.411445 22300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:29.411538 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a8ada5937bef43679cc3429ce96d2859"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.411609 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.420193 22300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.420610 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:29.420730 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.421131 22300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.421828 22300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.421872 22300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.421922 22300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.421950 22300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.429953 22300 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:39677
I20240502 14:00:29.430027 22427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:39677 every 8 connection(s)
I20240502 14:00:29.430874 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:00:29.438391 22428 heartbeater.cc:347] Connected to a master server at 127.16.217.62:43279
I20240502 14:00:29.438549 22428 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.438869 22428 heartbeater.cc:511] Master 127.16.217.62:43279 requested a full tablet report, sending...
I20240502 14:00:29.439620 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22300
I20240502 14:00:29.439733 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:00:29.440114 22240 ts_manager.cc:194] Registered new tserver with Master: a8ada5937bef43679cc3429ce96d2859 (127.16.217.1:39677)
I20240502 14:00:29.441179 22240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:39501
I20240502 14:00:29.441709 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:29.533084 22431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.533246 22431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.533286 22431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.535828 22431 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.535878 22431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.535961 22431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20240502 14:00:29.538964 22431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--never_fsync=true
--heap_profile_path=/tmp/kudu.22431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.539613 22431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:29.540474 22431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.546919 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.547425 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.547937 22437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.548676 22431 server_base.cc:1034] running on GCE node
I20240502 14:00:29.548974 22431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:29.549393 22431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:29.550529 22431 hybrid_clock.cc:648] HybridClock initialized: now 1714658429550511 us; error 33 us; skew 500 ppm
I20240502 14:00:29.552044 22431 webserver.cc:460] Webserver started at http://127.16.217.2:40965/ using document root <none> and password file <none>
I20240502 14:00:29.552523 22431 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.552577 22431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.552737 22431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.554009 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a4f44840fbc14fac98f927e6d1d99897"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.554392 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a4f44840fbc14fac98f927e6d1d99897"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.557039 22431 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 14:00:29.558825 22445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.559604 22431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:29.559696 22431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a4f44840fbc14fac98f927e6d1d99897"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.559760 22431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.583309 22431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.583719 22431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:29.583844 22431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.584229 22431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.584926 22431 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.584970 22431 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.585016 22431 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.585043 22431 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.592124 22431 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:34147
I20240502 14:00:29.592145 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:34147 every 8 connection(s)
I20240502 14:00:29.592847 22431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:00:29.596760 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22431
I20240502 14:00:29.596868 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:00:29.598778 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:29.600057 22559 heartbeater.cc:347] Connected to a master server at 127.16.217.62:43279
I20240502 14:00:29.600263 22559 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.600739 22559 heartbeater.cc:511] Master 127.16.217.62:43279 requested a full tablet report, sending...
I20240502 14:00:29.601559 22240 ts_manager.cc:194] Registered new tserver with Master: a4f44840fbc14fac98f927e6d1d99897 (127.16.217.2:34147)
I20240502 14:00:29.602047 22240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:57929
W20240502 14:00:29.691988 22562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.692199 22562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.692266 22562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.694787 22562 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.694844 22562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.694919 22562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20240502 14:00:29.697934 22562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--never_fsync=true
--heap_profile_path=/tmp/kudu.22562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.698577 22562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:29.699438 22562 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.706075 22567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.706821 22562 server_base.cc:1034] running on GCE node
W20240502 14:00:29.707314 22568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.707718 22570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.708016 22562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:29.708480 22562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:29.709635 22562 hybrid_clock.cc:648] HybridClock initialized: now 1714658429709619 us; error 26 us; skew 500 ppm
I20240502 14:00:29.711144 22562 webserver.cc:460] Webserver started at http://127.16.217.3:32843/ using document root <none> and password file <none>
I20240502 14:00:29.711576 22562 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.711622 22562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.711752 22562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.713100 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "58dd9301c92a4d33ac4cf31d66c98263"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.714990 22562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "58dd9301c92a4d33ac4cf31d66c98263"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.717687 22562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:29.719336 22576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.720055 22562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:29.720132 22562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "58dd9301c92a4d33ac4cf31d66c98263"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.720207 22562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.740790 22562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.741231 22562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:29.741356 22562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.741755 22562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.742542 22562 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.742583 22562 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.742620 22562 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.742647 22562 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.749862 22562 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:37695
I20240502 14:00:29.749881 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:37695 every 8 connection(s)
I20240502 14:00:29.750497 22562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:00:29.754048 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22562
I20240502 14:00:29.754118 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:00:29.755908 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.4:0
--local_ip_for_outbound_sockets=127.16.217.4
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:29.757277 22690 heartbeater.cc:347] Connected to a master server at 127.16.217.62:43279
I20240502 14:00:29.757436 22690 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.757752 22690 heartbeater.cc:511] Master 127.16.217.62:43279 requested a full tablet report, sending...
I20240502 14:00:29.758500 22240 ts_manager.cc:194] Registered new tserver with Master: 58dd9301c92a4d33ac4cf31d66c98263 (127.16.217.3:37695)
I20240502 14:00:29.759172 22240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:46819
W20240502 14:00:29.844769 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.844915 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.844949 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.847465 22693 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.847513 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.847569 22693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.4
I20240502 14:00:29.850630 22693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.4
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.851207 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:29.852025 22693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:29.858628 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.858702 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:29.859220 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:29.859935 22693 server_base.cc:1034] running on GCE node
I20240502 14:00:29.860219 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:29.860646 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:29.861781 22693 hybrid_clock.cc:648] HybridClock initialized: now 1714658429861758 us; error 36 us; skew 500 ppm
I20240502 14:00:29.863348 22693 webserver.cc:460] Webserver started at http://127.16.217.4:45769/ using document root <none> and password file <none>
I20240502 14:00:29.863826 22693 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:29.863874 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:29.864018 22693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:29.865393 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "08d82cf741ba4a06856388db4f9aa838"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.865770 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "08d82cf741ba4a06856388db4f9aa838"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.868441 22693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:00:29.870182 22707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.870965 22693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:29.871042 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "08d82cf741ba4a06856388db4f9aa838"
format_stamp: "Formatted at 2024-05-02 14:00:29 on dist-test-slave-7msf"
I20240502 14:00:29.871129 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:29.885623 22693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:29.885989 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:29.886106 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:29.886471 22693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:29.887094 22693 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:29.887135 22693 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.887179 22693 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:29.887203 22693 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:29.893896 22693 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.4:41519
I20240502 14:00:29.893914 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.4:41519 every 8 connection(s)
I20240502 14:00:29.894532 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:29.901003 22821 heartbeater.cc:347] Connected to a master server at 127.16.217.62:43279
I20240502 14:00:29.901027 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22693
I20240502 14:00:29.901165 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:00:29.901193 22821 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:29.901510 22821 heartbeater.cc:511] Master 127.16.217.62:43279 requested a full tablet report, sending...
I20240502 14:00:29.902298 22239 ts_manager.cc:194] Registered new tserver with Master: 08d82cf741ba4a06856388db4f9aa838 (127.16.217.4:41519)
I20240502 14:00:29.902827 22239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.4:39683
I20240502 14:00:29.902966 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:0
--local_ip_for_outbound_sockets=127.16.217.5
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--builtin_ntp_servers=127.16.217.20:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:29.992772 22824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:29.992926 22824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:29.992986 22824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:29.995523 22824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:29.995579 22824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:29.995654 22824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 14:00:29.998654 22824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:43279
--never_fsync=true
--heap_profile_path=/tmp/kudu.22824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:29.999229 22824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:30.000031 22824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:30.006321 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:30.006791 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:30.007406 22832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:30.008095 22824 server_base.cc:1034] running on GCE node
I20240502 14:00:30.008421 22824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:30.008850 22824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:30.009984 22824 hybrid_clock.cc:648] HybridClock initialized: now 1714658430009979 us; error 21 us; skew 500 ppm
I20240502 14:00:30.011461 22824 webserver.cc:460] Webserver started at http://127.16.217.5:46587/ using document root <none> and password file <none>
I20240502 14:00:30.011907 22824 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:30.011961 22824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:30.012101 22824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:30.013398 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "386cddfb3a604fe79bf62ecfdb0693ec"
format_stamp: "Formatted at 2024-05-02 14:00:30 on dist-test-slave-7msf"
I20240502 14:00:30.013796 22824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "386cddfb3a604fe79bf62ecfdb0693ec"
format_stamp: "Formatted at 2024-05-02 14:00:30 on dist-test-slave-7msf"
I20240502 14:00:30.016485 22824 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20240502 14:00:30.018112 22838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:30.018864 22824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:30.018939 22824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "386cddfb3a604fe79bf62ecfdb0693ec"
format_stamp: "Formatted at 2024-05-02 14:00:30 on dist-test-slave-7msf"
I20240502 14:00:30.019008 22824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:30.040320 22824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:30.040735 22824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:30.040859 22824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:30.041291 22824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:30.042055 22824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:30.042101 22824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:30.042140 22824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:30.042171 22824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:30.049235 22824 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:44517
I20240502 14:00:30.049269 22951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:44517 every 8 connection(s)
I20240502 14:00:30.049935 22824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:30.056655 22952 heartbeater.cc:347] Connected to a master server at 127.16.217.62:43279
I20240502 14:00:30.056810 22952 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:30.057134 22952 heartbeater.cc:511] Master 127.16.217.62:43279 requested a full tablet report, sending...
I20240502 14:00:30.057400 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 22824
I20240502 14:00:30.057504 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:00:30.057999 22239 ts_manager.cc:194] Registered new tserver with Master: 386cddfb3a604fe79bf62ecfdb0693ec (127.16.217.5:44517)
I20240502 14:00:30.058573 22239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:41901
I20240502 14:00:30.059351 17252 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:00:30.066318 22240 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41586:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:00:30.080578 22886 tablet_service.cc:1467] Processing CreateTablet for tablet c6683bd315b845f0839ebf44af0c8be1 (DEFAULT_TABLE table=TestTable [id=5a5c512225954174ae2f8140f508aa29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.081274 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6683bd315b845f0839ebf44af0c8be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.081318 22624 tablet_service.cc:1467] Processing CreateTablet for tablet c6683bd315b845f0839ebf44af0c8be1 (DEFAULT_TABLE table=TestTable [id=5a5c512225954174ae2f8140f508aa29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.081318 22362 tablet_service.cc:1467] Processing CreateTablet for tablet c6683bd315b845f0839ebf44af0c8be1 (DEFAULT_TABLE table=TestTable [id=5a5c512225954174ae2f8140f508aa29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:30.082165 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6683bd315b845f0839ebf44af0c8be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.082165 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6683bd315b845f0839ebf44af0c8be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:30.085422 22969 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Bootstrap starting.
I20240502 14:00:30.087131 22970 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Bootstrap starting.
I20240502 14:00:30.087654 22969 tablet_bootstrap.cc:654] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.087711 22971 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Bootstrap starting.
I20240502 14:00:30.088407 22969 log.cc:824] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:30.088899 22970 tablet_bootstrap.cc:654] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.089257 22971 tablet_bootstrap.cc:654] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:30.089545 22970 log.cc:824] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:30.089838 22971 log.cc:824] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:30.089952 22969 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: No bootstrap required, opened a new log
I20240502 14:00:30.090095 22969 ts_tablet_manager.cc:1367] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 14:00:30.090926 22970 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: No bootstrap required, opened a new log
I20240502 14:00:30.091112 22970 ts_tablet_manager.cc:1367] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:00:30.092787 22969 raft_consensus.cc:348] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.092957 22969 raft_consensus.cc:374] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.093004 22969 raft_consensus.cc:729] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386cddfb3a604fe79bf62ecfdb0693ec, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.093436 22971 tablet_bootstrap.cc:492] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: No bootstrap required, opened a new log
I20240502 14:00:30.093376 22969 consensus_queue.cc:260] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.093586 22971 ts_tablet_manager.cc:1367] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 14:00:30.093858 22970 raft_consensus.cc:348] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.094015 22970 raft_consensus.cc:374] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.094069 22970 raft_consensus.cc:729] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ada5937bef43679cc3429ce96d2859, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.094076 22969 ts_tablet_manager.cc:1398] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20240502 14:00:30.094354 22952 heartbeater.cc:503] Master 127.16.217.62:43279 was elected leader, sending a full tablet report...
I20240502 14:00:30.094398 22970 consensus_queue.cc:260] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.094938 22428 heartbeater.cc:503] Master 127.16.217.62:43279 was elected leader, sending a full tablet report...
I20240502 14:00:30.095140 22970 ts_tablet_manager.cc:1398] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:30.096545 22971 raft_consensus.cc:348] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.096719 22971 raft_consensus.cc:374] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:30.096774 22971 raft_consensus.cc:729] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dd9301c92a4d33ac4cf31d66c98263, State: Initialized, Role: FOLLOWER
I20240502 14:00:30.097134 22971 consensus_queue.cc:260] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.097703 22690 heartbeater.cc:503] Master 127.16.217.62:43279 was elected leader, sending a full tablet report...
I20240502 14:00:30.097796 22971 ts_tablet_manager.cc:1398] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
W20240502 14:00:30.182219 22429 tablet.cc:2369] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:30.251317 22691 tablet.cc:2369] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:30.300596 22953 tablet.cc:2369] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:30.357556 22976 raft_consensus.cc:482] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:30.357693 22976 raft_consensus.cc:504] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.358698 22976 leader_election.cc:288] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 386cddfb3a604fe79bf62ecfdb0693ec (127.16.217.5:44517), 58dd9301c92a4d33ac4cf31d66c98263 (127.16.217.3:37695)
I20240502 14:00:30.362584 22644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6683bd315b845f0839ebf44af0c8be1" candidate_uuid: "a8ada5937bef43679cc3429ce96d2859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dd9301c92a4d33ac4cf31d66c98263" is_pre_election: true
I20240502 14:00:30.362601 22906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6683bd315b845f0839ebf44af0c8be1" candidate_uuid: "a8ada5937bef43679cc3429ce96d2859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" is_pre_election: true
I20240502 14:00:30.362838 22644 raft_consensus.cc:2454] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8ada5937bef43679cc3429ce96d2859 in term 0.
I20240502 14:00:30.362838 22906 raft_consensus.cc:2454] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8ada5937bef43679cc3429ce96d2859 in term 0.
I20240502 14:00:30.363240 22315 leader_election.cc:302] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386cddfb3a604fe79bf62ecfdb0693ec, a8ada5937bef43679cc3429ce96d2859; no voters:
I20240502 14:00:30.363453 22976 raft_consensus.cc:2781] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:30.363514 22976 raft_consensus.cc:482] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:30.363565 22976 raft_consensus.cc:3037] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.364305 22976 raft_consensus.cc:504] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.364574 22976 leader_election.cc:288] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [CANDIDATE]: Term 1 election: Requested vote from peers 386cddfb3a604fe79bf62ecfdb0693ec (127.16.217.5:44517), 58dd9301c92a4d33ac4cf31d66c98263 (127.16.217.3:37695)
I20240502 14:00:30.364869 22906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6683bd315b845f0839ebf44af0c8be1" candidate_uuid: "a8ada5937bef43679cc3429ce96d2859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386cddfb3a604fe79bf62ecfdb0693ec"
I20240502 14:00:30.364960 22906 raft_consensus.cc:3037] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.365684 22906 raft_consensus.cc:2454] T c6683bd315b845f0839ebf44af0c8be1 P 386cddfb3a604fe79bf62ecfdb0693ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ada5937bef43679cc3429ce96d2859 in term 1.
I20240502 14:00:30.365914 22644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6683bd315b845f0839ebf44af0c8be1" candidate_uuid: "a8ada5937bef43679cc3429ce96d2859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dd9301c92a4d33ac4cf31d66c98263"
I20240502 14:00:30.365962 22315 leader_election.cc:302] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386cddfb3a604fe79bf62ecfdb0693ec, a8ada5937bef43679cc3429ce96d2859; no voters:
I20240502 14:00:30.366022 22644 raft_consensus.cc:3037] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:30.366101 22976 raft_consensus.cc:2781] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:30.366293 22976 raft_consensus.cc:686] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [term 1 LEADER]: Becoming Leader. State: Replica: a8ada5937bef43679cc3429ce96d2859, State: Running, Role: LEADER
I20240502 14:00:30.366559 22976 consensus_queue.cc:237] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } }
I20240502 14:00:30.366816 22644 raft_consensus.cc:2454] T c6683bd315b845f0839ebf44af0c8be1 P 58dd9301c92a4d33ac4cf31d66c98263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8ada5937bef43679cc3429ce96d2859 in term 1.
I20240502 14:00:30.368608 22239 catalog_manager.cc:5484] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8ada5937bef43679cc3429ce96d2859 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a8ada5937bef43679cc3429ce96d2859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 } } peers { permanent_uuid: "a8ada5937bef43679cc3429ce96d2859" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 39677 } } peers { permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 } } }
I20240502 14:00:30.432773 17252 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:00:30.434163 17252 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a8ada5937bef43679cc3429ce96d2859 to finish bootstrapping
I20240502 14:00:30.439026 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a4f44840fbc14fac98f927e6d1d99897 to finish bootstrapping
I20240502 14:00:30.443593 17252 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 58dd9301c92a4d33ac4cf31d66c98263 to finish bootstrapping
I20240502 14:00:30.446987 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 08d82cf741ba4a06856388db4f9aa838 to finish bootstrapping
I20240502 14:00:30.451390 17252 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 386cddfb3a604fe79bf62ecfdb0693ec to finish bootstrapping
I20240502 14:00:30.603996 22559 heartbeater.cc:503] Master 127.16.217.62:43279 was elected leader, sending a full tablet report...
I20240502 14:00:30.607158 22755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.608012 22624 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.610401 22886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.612680 22362 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.627249 22493 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.689766 22886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.690951 22493 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.699299 22755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.699499 22624 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.699796 22362 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.877274 22755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.877969 22493 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.878340 22624 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.879323 22362 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.880090 22886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.904323 22821 heartbeater.cc:503] Master 127.16.217.62:43279 was elected leader, sending a full tablet report...
I20240502 14:00:30.943997 23109 consensus_queue.cc:1035] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dd9301c92a4d33ac4cf31d66c98263" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.949141 23114 consensus_queue.cc:1035] T c6683bd315b845f0839ebf44af0c8be1 P a8ada5937bef43679cc3429ce96d2859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386cddfb3a604fe79bf62ecfdb0693ec" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:30.957070 22755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.959301 22362 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:30.960079 22624 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.960361 22493 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:30.962507 22886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:31.045974 22624 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:31.046008 22886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:31.046301 22493 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:31.049124 22362 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:31.056845 22755 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:31.084328 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22300
I20240502 14:00:31.089979 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22431
I20240502 14:00:31.095031 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22562
I20240502 14:00:31.099973 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22693
I20240502 14:00:31.104879 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22824
I20240502 14:00:31.109899 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 22209
2024-05-02T14:00:31Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2061 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2061 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 14:00:31.131778 17252 test_util.cc:268] Using random seed: 112386174
I20240502 14:00:31.132613 17252 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:00:31.132652 17252 ts_itest-base.cc:113] --------------
I20240502 14:00:31.132664 17252 ts_itest-base.cc:114] 7 tablet servers
I20240502 14:00:31.132676 17252 ts_itest-base.cc:115] 2 replicas per TS
I20240502 14:00:31.132704 17252 ts_itest-base.cc:116] --------------
2024-05-02T14:00:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:00:31Z Disabled control of system clock
I20240502 14:00:31.139668 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32783
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:32783
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 14:00:31.229239 23166 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:00:31.229413 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.229461 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:31.229521 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.232033 23166 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:31.232095 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:00:31.232146 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:31.232183 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:00:31.232223 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:00:31.235172 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:32783
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.23166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:31.235812 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:31.236694 23166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:31.242671 23174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.243139 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.243288 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:31.243963 23166 server_base.cc:1034] running on GCE node
I20240502 14:00:31.244419 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:31.245102 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:31.246241 23166 hybrid_clock.cc:648] HybridClock initialized: now 1714658431246222 us; error 31 us; skew 500 ppm
I20240502 14:00:31.247639 23166 webserver.cc:460] Webserver started at http://127.16.217.62:45721/ using document root <none> and password file <none>
I20240502 14:00:31.248080 23166 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:31.248131 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:31.248324 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:31.249605 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c7eeb47ae899472f8cb9c37a97c5c29d"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.250000 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c7eeb47ae899472f8cb9c37a97c5c29d"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.252765 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:00:31.254455 23180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.255236 23166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.255331 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c7eeb47ae899472f8cb9c37a97c5c29d"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.255398 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:31.268040 23166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:31.268548 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:31.268672 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:31.274004 23166 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:32783
I20240502 14:00:31.274005 23232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:32783 every 8 connection(s)
I20240502 14:00:31.274718 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:00:31.276682 23233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:31.280661 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d: Bootstrap starting.
I20240502 14:00:31.282444 23233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:31.282953 23233 log.cc:824] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:31.283408 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23166
I20240502 14:00:31.283515 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:00:31.284457 23233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d: No bootstrap required, opened a new log
I20240502 14:00:31.287088 23233 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } }
I20240502 14:00:31.287226 23233 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:31.287277 23233 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7eeb47ae899472f8cb9c37a97c5c29d, State: Initialized, Role: FOLLOWER
I20240502 14:00:31.287600 23233 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } }
I20240502 14:00:31.287708 23233 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:00:31.287792 23233 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:00:31.287896 23233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:31.288664 23233 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } }
I20240502 14:00:31.288954 23233 leader_election.cc:302] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7eeb47ae899472f8cb9c37a97c5c29d; no voters:
I20240502 14:00:31.289220 23233 leader_election.cc:288] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:00:31.289546 23238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:31.289738 23238 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [term 1 LEADER]: Becoming Leader. State: Replica: c7eeb47ae899472f8cb9c37a97c5c29d, State: Running, Role: LEADER
I20240502 14:00:31.290019 23238 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } }
I20240502 14:00:31.290236 23233 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:00:31.291019 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7eeb47ae899472f8cb9c37a97c5c29d. Latest consensus state: current_term: 1 leader_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } } }
I20240502 14:00:31.291127 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:31.291563 23246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:00:31.291910 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7eeb47ae899472f8cb9c37a97c5c29d" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 32783 } } }
I20240502 14:00:31.292074 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d [sys.catalog]: This master's current role is: LEADER
I20240502 14:00:31.294119 23246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:00:31.298250 23246 catalog_manager.cc:1347] Generated new cluster ID: 8234001b025444079492ea8cfa9c33bb
I20240502 14:00:31.298297 23246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:00:31.306053 23246 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:00:31.306695 23246 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:00:31.315914 23246 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c7eeb47ae899472f8cb9c37a97c5c29d: Generated new TSK 0
I20240502 14:00:31.316469 23246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:00:31.324672 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:31.418246 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.418414 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:31.418480 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.421042 23257 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:31.421098 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:31.421214 23257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20240502 14:00:31.424243 23257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:31.424887 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:31.425747 23257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:31.432607 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.432716 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.433158 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:31.433898 23257 server_base.cc:1034] running on GCE node
I20240502 14:00:31.434180 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:31.434592 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:31.435731 23257 hybrid_clock.cc:648] HybridClock initialized: now 1714658431435715 us; error 33 us; skew 500 ppm
I20240502 14:00:31.437217 23257 webserver.cc:460] Webserver started at http://127.16.217.1:45437/ using document root <none> and password file <none>
I20240502 14:00:31.437664 23257 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:31.437722 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:31.437896 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:31.439260 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6395860ed21149cab200c8364faf6fa8"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.439677 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6395860ed21149cab200c8364faf6fa8"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.442576 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:00:31.444413 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.445214 23257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.445309 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6395860ed21149cab200c8364faf6fa8"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.445382 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:31.458014 23257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:31.458482 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:31.458608 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:31.459002 23257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:31.459703 23257 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:31.459750 23257 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.459801 23257 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:31.459827 23257 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.467537 23257 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:35121
I20240502 14:00:31.467553 23384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:35121 every 8 connection(s)
I20240502 14:00:31.468350 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:00:31.471076 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23257
I20240502 14:00:31.471267 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:00:31.473228 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:31.475514 23385 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:31.475705 23385 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:31.476044 23385 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:31.477268 23197 ts_manager.cc:194] Registered new tserver with Master: 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:31.478248 23197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:44801
W20240502 14:00:31.565533 23388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.565682 23388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:31.565740 23388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.568249 23388 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:31.568327 23388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:31.568387 23388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20240502 14:00:31.571277 23388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:31.571856 23388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:31.572721 23388 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:31.578902 23396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.579435 23393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.579917 23394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:31.580668 23388 server_base.cc:1034] running on GCE node
I20240502 14:00:31.580950 23388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:31.581373 23388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:31.582516 23388 hybrid_clock.cc:648] HybridClock initialized: now 1714658431582496 us; error 32 us; skew 500 ppm
I20240502 14:00:31.583930 23388 webserver.cc:460] Webserver started at http://127.16.217.2:33185/ using document root <none> and password file <none>
I20240502 14:00:31.584410 23388 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:31.584465 23388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:31.584614 23388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:31.585868 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "49daee87c26c4b989eaec212705a11f9"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.586249 23388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "49daee87c26c4b989eaec212705a11f9"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.588948 23388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:00:31.590934 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.591773 23388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:31.591871 23388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "49daee87c26c4b989eaec212705a11f9"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.591950 23388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:31.619110 23388 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:31.619566 23388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:31.619691 23388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:31.620086 23388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:31.620805 23388 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:31.620849 23388 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.620896 23388 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:31.620922 23388 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.628043 23388 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:36885
I20240502 14:00:31.628062 23515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:36885 every 8 connection(s)
I20240502 14:00:31.628794 23388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:00:31.635823 23516 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:31.635968 23516 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:31.636262 23516 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:31.637033 23197 ts_manager.cc:194] Registered new tserver with Master: 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:31.637526 23197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:41831
I20240502 14:00:31.638310 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23388
I20240502 14:00:31.638409 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:00:31.640085 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:31.729986 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.730156 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:31.730221 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.732856 23519 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:31.732908 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:31.732996 23519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20240502 14:00:31.735877 23519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:31.736539 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:31.737391 23519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:31.744025 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.744102 23525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.744580 23527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:31.745307 23519 server_base.cc:1034] running on GCE node
I20240502 14:00:31.745564 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:31.745949 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:31.747072 23519 hybrid_clock.cc:648] HybridClock initialized: now 1714658431747052 us; error 31 us; skew 500 ppm
I20240502 14:00:31.748587 23519 webserver.cc:460] Webserver started at http://127.16.217.3:46839/ using document root <none> and password file <none>
I20240502 14:00:31.749027 23519 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:31.749090 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:31.749241 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:31.750486 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "968c2f6eed3842dfa034660fa0a2117b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.750862 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "968c2f6eed3842dfa034660fa0a2117b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.753712 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:00:31.755450 23533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.756227 23519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:31.756327 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "968c2f6eed3842dfa034660fa0a2117b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.756397 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:31.770313 23519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:31.770731 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:31.770860 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:31.771261 23519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:31.771973 23519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:31.772018 23519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.772068 23519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:31.772096 23519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.779361 23519 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:33339
I20240502 14:00:31.779381 23646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:33339 every 8 connection(s)
I20240502 14:00:31.780005 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:00:31.784641 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23519
I20240502 14:00:31.784739 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:00:31.786600 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.4:0
--local_ip_for_outbound_sockets=127.16.217.4
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:31.786645 23647 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:31.786813 23647 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:31.787144 23647 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:31.787942 23197 ts_manager.cc:194] Registered new tserver with Master: 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:31.788549 23197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:48241
W20240502 14:00:31.877488 23650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:31.877630 23650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:31.877666 23650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:31.880203 23650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:31.880249 23650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:31.880334 23650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.4
I20240502 14:00:31.883489 23650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.4
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:31.884079 23650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:31.884938 23650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:31.891002 23655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.891551 23658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:31.892027 23656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:31.892760 23650 server_base.cc:1034] running on GCE node
I20240502 14:00:31.893007 23650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:31.893386 23650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:31.894527 23650 hybrid_clock.cc:648] HybridClock initialized: now 1714658431894513 us; error 34 us; skew 500 ppm
I20240502 14:00:31.895985 23650 webserver.cc:460] Webserver started at http://127.16.217.4:35743/ using document root <none> and password file <none>
I20240502 14:00:31.896528 23650 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:31.896589 23650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:31.896780 23650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:31.898030 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "916fb151c5274f2c8caed395dcdf412b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.898430 23650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "916fb151c5274f2c8caed395dcdf412b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.900962 23650 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:00:31.902735 23664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.903446 23650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:31.903519 23650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "916fb151c5274f2c8caed395dcdf412b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:31.903573 23650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:31.924662 23650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:31.925065 23650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:31.925176 23650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:31.925541 23650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:31.926192 23650 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:31.926234 23650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.926263 23650 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:31.926283 23650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:31.933517 23650 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.4:41841
I20240502 14:00:31.933538 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.4:41841 every 8 connection(s)
I20240502 14:00:31.934186 23650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:31.940557 23778 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:31.940730 23778 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:31.941041 23778 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:31.941629 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23650
I20240502 14:00:31.941731 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:00:31.941893 23197 ts_manager.cc:194] Registered new tserver with Master: 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841)
I20240502 14:00:31.942363 23197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.4:42953
I20240502 14:00:31.943646 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:0
--local_ip_for_outbound_sockets=127.16.217.5
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:32.034582 23781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:32.034739 23781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:32.034781 23781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:32.037398 23781 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:32.037444 23781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:32.037518 23781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 14:00:32.040431 23781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:32.041002 23781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:32.041896 23781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:32.048408 23787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.048898 23786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.049373 23789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:32.050081 23781 server_base.cc:1034] running on GCE node
I20240502 14:00:32.050385 23781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:32.050788 23781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:32.051919 23781 hybrid_clock.cc:648] HybridClock initialized: now 1714658432051913 us; error 23 us; skew 500 ppm
I20240502 14:00:32.053320 23781 webserver.cc:460] Webserver started at http://127.16.217.5:44519/ using document root <none> and password file <none>
I20240502 14:00:32.053732 23781 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:32.053788 23781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:32.053923 23781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:32.055183 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "17a03815fc864e4db7a68c32c6e42616"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.055545 23781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "17a03815fc864e4db7a68c32c6e42616"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.058281 23781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 14:00:32.060006 23795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.060791 23781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:00:32.060873 23781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "17a03815fc864e4db7a68c32c6e42616"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.060940 23781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:32.075863 23781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:32.076285 23781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:32.076400 23781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:32.076767 23781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:32.077402 23781 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:32.077443 23781 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.077486 23781 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:32.077512 23781 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.084371 23781 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:43149
I20240502 14:00:32.084407 23908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:43149 every 8 connection(s)
I20240502 14:00:32.085004 23781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:32.088071 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23781
I20240502 14:00:32.088169 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:00:32.090029 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:0
--local_ip_for_outbound_sockets=127.16.217.6
--webserver_interface=127.16.217.6
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:32.091636 23909 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:32.091807 23909 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:32.092135 23909 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:32.092868 23196 ts_manager.cc:194] Registered new tserver with Master: 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149)
I20240502 14:00:32.093335 23196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:50287
W20240502 14:00:32.177966 23912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:32.178107 23912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:32.178164 23912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:32.180600 23912 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:32.180652 23912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:32.180721 23912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 14:00:32.183774 23912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.23912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:32.184396 23912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:32.185200 23912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:32.191110 23918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.191663 23917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.192168 23920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:32.192893 23912 server_base.cc:1034] running on GCE node
I20240502 14:00:32.193130 23912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:32.193498 23912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:32.194622 23912 hybrid_clock.cc:648] HybridClock initialized: now 1714658432194604 us; error 27 us; skew 500 ppm
I20240502 14:00:32.196100 23912 webserver.cc:460] Webserver started at http://127.16.217.6:41419/ using document root <none> and password file <none>
I20240502 14:00:32.196529 23912 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:32.196578 23912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:32.196698 23912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:32.197877 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "66c6a4ade9e84186beea567c21f80eb8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.198236 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "66c6a4ade9e84186beea567c21f80eb8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.200680 23912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 14:00:32.202384 23926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.203147 23912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:32.203222 23912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "66c6a4ade9e84186beea567c21f80eb8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.203280 23912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:32.221798 23912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:32.222225 23912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:32.222332 23912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:32.222693 23912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:32.223301 23912 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:32.223340 23912 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.223367 23912 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:32.223387 23912 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.230249 23912 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:42367
I20240502 14:00:32.230289 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:42367 every 8 connection(s)
I20240502 14:00:32.230878 23912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:32.234651 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 23912
I20240502 14:00:32.234747 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:00:32.236588 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:0
--local_ip_for_outbound_sockets=127.16.217.7
--webserver_interface=127.16.217.7
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:32.237648 24040 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:32.237795 24040 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:32.238193 24040 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:32.239006 23196 ts_manager.cc:194] Registered new tserver with Master: 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367)
I20240502 14:00:32.239468 23196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:49149
W20240502 14:00:32.324949 24043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:32.325091 24043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:32.325153 24043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:32.327673 24043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:32.327728 24043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:32.327803 24043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 14:00:32.330788 24043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.24043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:32.331450 24043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:32.332357 24043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:32.339005 24051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.339020 24048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:32.339541 24049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:32.340226 24043 server_base.cc:1034] running on GCE node
I20240502 14:00:32.340493 24043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:32.340901 24043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:32.342041 24043 hybrid_clock.cc:648] HybridClock initialized: now 1714658432342031 us; error 24 us; skew 500 ppm
I20240502 14:00:32.343428 24043 webserver.cc:460] Webserver started at http://127.16.217.7:39889/ using document root <none> and password file <none>
I20240502 14:00:32.343842 24043 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:32.343888 24043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:32.344031 24043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:00:32.345278 24043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a3101d9789354f17b9af4da59878eec8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.345651 24043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a3101d9789354f17b9af4da59878eec8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.348186 24043 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:00:32.349956 24057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.350683 24043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:32.350764 24043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a3101d9789354f17b9af4da59878eec8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:32.350831 24043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:32.371291 24043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:32.371744 24043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:32.371870 24043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:32.372292 24043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:32.372997 24043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:32.373036 24043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.373070 24043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:32.373099 24043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:32.379907 24043 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:32823
I20240502 14:00:32.379927 24170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:32823 every 8 connection(s)
I20240502 14:00:32.380571 24043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:32.381451 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 24043
I20240502 14:00:32.381552 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:00:32.387463 24171 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:32.387617 24171 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:32.387953 24171 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:32.388696 23196 ts_manager.cc:194] Registered new tserver with Master: a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:32.389164 23196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:37825
I20240502 14:00:32.394886 17252 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:00:32.402043 23196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:00:32.415194 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb5ab6a504d4fceab781c331a40627d (DEFAULT_TABLE table=TestTable [id=bb67e7032177475e90df07c57c0fe276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.415196 23712 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb5ab6a504d4fceab781c331a40627d (DEFAULT_TABLE table=TestTable [id=bb67e7032177475e90df07c57c0fe276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:32.415803 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb5ab6a504d4fceab781c331a40627d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.415810 23712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb5ab6a504d4fceab781c331a40627d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:32.419556 24187 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Bootstrap starting.
I20240502 14:00:32.419570 24188 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:32.421226 24188 tablet_bootstrap.cc:654] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.421263 24187 tablet_bootstrap.cc:654] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:32.421938 24187 log.cc:824] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:32.421940 24188 log.cc:824] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:32.423125 24188 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: No bootstrap required, opened a new log
I20240502 14:00:32.423132 24187 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: No bootstrap required, opened a new log
I20240502 14:00:32.423240 24187 ts_tablet_manager.cc:1367] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:00:32.423247 24188 ts_tablet_manager.cc:1367] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:00:32.425410 24187 raft_consensus.cc:348] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.425422 24188 raft_consensus.cc:348] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.425536 24187 raft_consensus.cc:374] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.425539 24188 raft_consensus.cc:374] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:32.425565 24187 raft_consensus.cc:729] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.425582 24188 raft_consensus.cc:729] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:32.425879 24187 consensus_queue.cc:260] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.425885 24188 consensus_queue.cc:260] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.426515 24187 ts_tablet_manager.cc:1398] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 14:00:32.426519 24188 ts_tablet_manager.cc:1398] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 14:00:32.426766 23778 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:32.426776 24171 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
W20240502 14:00:32.434950 23779 tablet.cc:2369] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:32.479948 23385 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:32.604470 24192 raft_consensus.cc:482] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:32.604631 24192 raft_consensus.cc:504] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.605469 24192 leader_election.cc:288] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:32.608686 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5ab6a504d4fceab781c331a40627d" candidate_uuid: "916fb151c5274f2c8caed395dcdf412b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3101d9789354f17b9af4da59878eec8" is_pre_election: true
I20240502 14:00:32.608927 24125 raft_consensus.cc:2454] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916fb151c5274f2c8caed395dcdf412b in term 0.
I20240502 14:00:32.609282 23667 leader_election.cc:302] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 916fb151c5274f2c8caed395dcdf412b, a3101d9789354f17b9af4da59878eec8; no voters:
I20240502 14:00:32.609493 24192 raft_consensus.cc:2781] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:32.609555 24192 raft_consensus.cc:482] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:32.609593 24192 raft_consensus.cc:3037] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.610380 24192 raft_consensus.cc:504] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.610606 24192 leader_election.cc:288] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 1 election: Requested vote from peers a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:32.610872 24125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5ab6a504d4fceab781c331a40627d" candidate_uuid: "916fb151c5274f2c8caed395dcdf412b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3101d9789354f17b9af4da59878eec8"
I20240502 14:00:32.610952 24125 raft_consensus.cc:3037] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:32.611752 24125 raft_consensus.cc:2454] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916fb151c5274f2c8caed395dcdf412b in term 1.
I20240502 14:00:32.611979 23667 leader_election.cc:302] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 916fb151c5274f2c8caed395dcdf412b, a3101d9789354f17b9af4da59878eec8; no voters:
I20240502 14:00:32.612123 24192 raft_consensus.cc:2781] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:32.612357 24192 raft_consensus.cc:686] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 LEADER]: Becoming Leader. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Running, Role: LEADER
I20240502 14:00:32.612823 24192 consensus_queue.cc:237] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:32.615092 23197 catalog_manager.cc:5484] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b reported cstate change: term changed from 0 to 1, leader changed from <none> to 916fb151c5274f2c8caed395dcdf412b (127.16.217.4). New cstate: current_term: 1 leader_uuid: "916fb151c5274f2c8caed395dcdf412b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
W20240502 14:00:32.631201 24172 tablet.cc:2369] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:32.639073 23516 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:32.640246 17252 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:00:32.641345 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6395860ed21149cab200c8364faf6fa8 to finish bootstrapping
I20240502 14:00:32.646134 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 49daee87c26c4b989eaec212705a11f9 to finish bootstrapping
I20240502 14:00:32.650641 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 968c2f6eed3842dfa034660fa0a2117b to finish bootstrapping
I20240502 14:00:32.654924 17252 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 916fb151c5274f2c8caed395dcdf412b to finish bootstrapping
I20240502 14:00:32.658620 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 17a03815fc864e4db7a68c32c6e42616 to finish bootstrapping
I20240502 14:00:32.662954 17252 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 66c6a4ade9e84186beea567c21f80eb8 to finish bootstrapping
I20240502 14:00:32.667037 17252 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a3101d9789354f17b9af4da59878eec8 to finish bootstrapping
I20240502 14:00:32.670610 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23650
I20240502 14:00:32.675977 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23781
I20240502 14:00:32.680507 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23912
I20240502 14:00:32.685051 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 24043
I20240502 14:00:32.790132 23647 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:36.441430 23197 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:36.454202 23319 tablet_service.cc:1467] Processing CreateTablet for tablet 87edafbabb594a40b7b20ea97a6e8e51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.454295 23581 tablet_service.cc:1467] Processing CreateTablet for tablet 87edafbabb594a40b7b20ea97a6e8e51 (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.454450 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 0b935eeb4b2344d78780f44fff524e9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.454450 23450 tablet_service.cc:1467] Processing CreateTablet for tablet 9b29f51fa8f14e3bb8b859b1941e6fac (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.455104 23319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87edafbabb594a40b7b20ea97a6e8e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.455104 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87edafbabb594a40b7b20ea97a6e8e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.455318 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b29f51fa8f14e3bb8b859b1941e6fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.455535 23580 tablet_service.cc:1467] Processing CreateTablet for tablet 0b935eeb4b2344d78780f44fff524e9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.455797 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935eeb4b2344d78780f44fff524e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.456248 23318 tablet_service.cc:1467] Processing CreateTablet for tablet 9b29f51fa8f14e3bb8b859b1941e6fac (DEFAULT_TABLE table=rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:36.456523 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b29f51fa8f14e3bb8b859b1941e6fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.459321 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935eeb4b2344d78780f44fff524e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:36.461041 24214 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:36.461364 24213 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:36.462908 24212 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:36.462963 24214 tablet_bootstrap.cc:654] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.463338 24213 tablet_bootstrap.cc:654] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.463618 24214 log.cc:824] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.464211 24213 log.cc:824] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.464840 24212 tablet_bootstrap.cc:654] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.465431 24214 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:36.465540 24212 log.cc:824] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:36.465582 24214 ts_tablet_manager.cc:1367] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:00:36.465662 24213 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:36.465818 24213 ts_tablet_manager.cc:1367] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:00:36.466806 24212 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:36.466967 24212 ts_tablet_manager.cc:1367] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:00:36.469163 24214 raft_consensus.cc:348] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.469163 24213 raft_consensus.cc:348] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.469345 24214 raft_consensus.cc:374] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.469345 24213 raft_consensus.cc:374] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.469391 24214 raft_consensus.cc:729] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.469399 24213 raft_consensus.cc:729] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.469807 24214 consensus_queue.cc:260] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.469807 24213 consensus_queue.cc:260] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.470553 24214 ts_tablet_manager.cc:1398] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:00:36.470553 24213 ts_tablet_manager.cc:1398] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 14:00:36.470677 24212 raft_consensus.cc:348] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.470824 24212 raft_consensus.cc:374] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.470853 24213 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:36.470853 24214 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:36.470862 24212 raft_consensus.cc:729] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.471171 24212 consensus_queue.cc:260] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.471668 24214 tablet_bootstrap.cc:654] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.471668 24213 tablet_bootstrap.cc:654] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.471869 24212 ts_tablet_manager.cc:1398] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20240502 14:00:36.472498 23386 tablet.cc:2369] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:36.473057 24212 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:36.474407 24212 tablet_bootstrap.cc:654] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:36.475435 24212 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:36.475508 24212 ts_tablet_manager.cc:1367] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:36.475939 24212 raft_consensus.cc:348] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.476014 24212 raft_consensus.cc:374] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.476045 24212 raft_consensus.cc:729] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.476181 24212 consensus_queue.cc:260] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.477089 24214 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:36.477190 24214 ts_tablet_manager.cc:1367] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:00:36.477247 24213 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:36.477342 24213 ts_tablet_manager.cc:1367] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:00:36.477757 24214 raft_consensus.cc:348] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.477805 24213 raft_consensus.cc:348] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.477867 24214 raft_consensus.cc:374] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.477903 24213 raft_consensus.cc:374] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:36.477944 24213 raft_consensus.cc:729] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.477936 24214 raft_consensus.cc:729] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:36.477988 24212 ts_tablet_manager.cc:1398] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:00:36.478047 24213 consensus_queue.cc:260] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.478107 24214 consensus_queue.cc:260] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.478297 24213 ts_tablet_manager.cc:1398] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:00:36.478431 24214 ts_tablet_manager.cc:1398] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:36.480944 24219 raft_consensus.cc:482] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.481063 24219 raft_consensus.cc:504] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.481802 24219 leader_election.cc:288] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:36.484746 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8" is_pre_election: true
I20240502 14:00:36.484941 23339 raft_consensus.cc:2454] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 0.
I20240502 14:00:36.485271 23404 leader_election.cc:302] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:36.485447 24219 raft_consensus.cc:2781] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.485504 24219 raft_consensus.cc:482] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.485529 24219 raft_consensus.cc:3037] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.486238 24219 raft_consensus.cc:504] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.486476 24219 leader_election.cc:288] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:36.486817 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:36.486901 23339 raft_consensus.cc:3037] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.487651 23339 raft_consensus.cc:2454] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 1.
I20240502 14:00:36.487944 23404 leader_election.cc:302] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:36.488111 24219 raft_consensus.cc:2781] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.488328 24219 raft_consensus.cc:686] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Becoming Leader. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:36.488562 24219 consensus_queue.cc:237] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.490774 23195 catalog_manager.cc:5484] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49daee87c26c4b989eaec212705a11f9 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
I20240502 14:00:36.532137 24220 raft_consensus.cc:482] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.532358 24220 raft_consensus.cc:504] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.533465 24220 leader_election.cc:288] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
W20240502 14:00:36.533767 23648 tablet.cc:2369] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:36.536762 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87edafbabb594a40b7b20ea97a6e8e51" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8" is_pre_election: true
I20240502 14:00:36.536873 23339 raft_consensus.cc:2454] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 0.
I20240502 14:00:36.537225 23535 leader_election.cc:302] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:36.537433 24220 raft_consensus.cc:2781] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.537496 24220 raft_consensus.cc:482] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.537552 24220 raft_consensus.cc:3037] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.538306 24220 raft_consensus.cc:504] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.538507 24220 leader_election.cc:288] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:36.538830 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87edafbabb594a40b7b20ea97a6e8e51" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:36.538916 23339 raft_consensus.cc:3037] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.539580 23339 raft_consensus.cc:2454] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 1.
I20240502 14:00:36.539842 23535 leader_election.cc:302] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:36.540002 24220 raft_consensus.cc:2781] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.540089 24220 raft_consensus.cc:686] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Becoming Leader. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Running, Role: LEADER
I20240502 14:00:36.540393 24220 consensus_queue.cc:237] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:36.542623 23195 catalog_manager.cc:5484] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3). New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
W20240502 14:00:36.632793 23517 tablet.cc:2369] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:36.823912 24219 raft_consensus.cc:482] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:36.824025 24219 raft_consensus.cc:504] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.824399 24219 leader_election.cc:288] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:36.827476 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935eeb4b2344d78780f44fff524e9d" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:36.827672 23601 raft_consensus.cc:2454] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 0.
I20240502 14:00:36.827992 23403 leader_election.cc:302] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:36.828191 24219 raft_consensus.cc:2781] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:36.828258 24219 raft_consensus.cc:482] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:36.828315 24219 raft_consensus.cc:3037] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.829118 24219 raft_consensus.cc:504] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.829392 24219 leader_election.cc:288] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:36.829728 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935eeb4b2344d78780f44fff524e9d" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:36.829828 23601 raft_consensus.cc:3037] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:36.830578 23601 raft_consensus.cc:2454] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 1.
I20240502 14:00:36.830875 23403 leader_election.cc:302] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:36.831041 24219 raft_consensus.cc:2781] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:36.831125 24219 raft_consensus.cc:686] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Becoming Leader. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:36.831251 24219 consensus_queue.cc:237] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:36.833002 23195 catalog_manager.cc:5484] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49daee87c26c4b989eaec212705a11f9 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } }
I20240502 14:00:36.971194 24219 consensus_queue.cc:1035] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:36.995239 24231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:36.995430 24231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:36.998275 24231 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:00:37.059588 24261 consensus_queue.cc:1035] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.067730 23601 raft_consensus.cc:1261] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.068528 24226 consensus_queue.cc:1035] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.086913 24266 mvcc.cc:205] Tried to move back new op lower bound from 7023240958222872576 to 7023240957261701120. Current Snapshot: MvccSnapshot[applied={T|T < 7023240958222872576}]
I20240502 14:00:37.086947 24267 mvcc.cc:205] Tried to move back new op lower bound from 7023240958222872576 to 7023240957261701120. Current Snapshot: MvccSnapshot[applied={T|T < 7023240958222872576}]
INSERT report
rows total: 2000
time total: 74.5535 ms
time per row: 0.0372767 ms
I20240502 14:00:37.108918 23195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:37.113727 23318 tablet_service.cc:1467] Processing CreateTablet for tablet dbbf1e7447604baa9701d6330ff31a3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.114009 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf1e7447604baa9701d6330ff31a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.114050 23581 tablet_service.cc:1467] Processing CreateTablet for tablet b043771f4c844e86a8d7111f19c45389 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.114091 23450 tablet_service.cc:1467] Processing CreateTablet for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.114336 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043771f4c844e86a8d7111f19c45389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.114346 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.114751 23580 tablet_service.cc:1467] Processing CreateTablet for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.114935 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.116421 23449 tablet_service.cc:1467] Processing CreateTablet for tablet dbbf1e7447604baa9701d6330ff31a3f (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.116499 24284 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:37.116632 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf1e7447604baa9701d6330ff31a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.116856 24283 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:37.117362 24284 tablet_bootstrap.cc:654] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.117363 24282 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:37.117722 24283 tablet_bootstrap.cc:654] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.118166 24282 tablet_bootstrap.cc:654] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.118556 23318 tablet_service.cc:1467] Processing CreateTablet for tablet b043771f4c844e86a8d7111f19c45389 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.118782 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043771f4c844e86a8d7111f19c45389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.118970 24283 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:37.119050 24283 ts_tablet_manager.cc:1367] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:37.119392 24284 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:37.119480 24284 ts_tablet_manager.cc:1367] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:37.119450 24283 raft_consensus.cc:348] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.119544 24283 raft_consensus.cc:374] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.119575 24283 raft_consensus.cc:729] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.119699 24283 consensus_queue.cc:260] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.119813 24282 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:37.119881 24282 ts_tablet_manager.cc:1367] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:37.119889 24284 raft_consensus.cc:348] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.119959 24283 ts_tablet_manager.cc:1398] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.119992 24284 raft_consensus.cc:374] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.120029 24284 raft_consensus.cc:729] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.120115 24283 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:37.120136 24284 consensus_queue.cc:260] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.120355 24282 raft_consensus.cc:348] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.120417 24284 ts_tablet_manager.cc:1398] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.120445 24282 raft_consensus.cc:374] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.120476 24282 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.120572 24284 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:37.120584 24282 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.120817 24282 ts_tablet_manager.cc:1398] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.120936 24283 tablet_bootstrap.cc:654] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.121076 24282 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:37.121351 24284 tablet_bootstrap.cc:654] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.121973 24282 tablet_bootstrap.cc:654] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.122006 24283 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:37.122067 24283 ts_tablet_manager.cc:1367] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:37.122495 24283 raft_consensus.cc:348] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.122579 24283 raft_consensus.cc:374] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.122618 24283 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.122740 24283 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.123013 24283 ts_tablet_manager.cc:1398] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.123261 24282 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:37.123327 24282 ts_tablet_manager.cc:1367] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:37.123646 24282 raft_consensus.cc:348] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.123734 24282 raft_consensus.cc:374] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.123775 24282 raft_consensus.cc:729] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.123880 24282 consensus_queue.cc:260] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.124112 24282 ts_tablet_manager.cc:1398] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.124387 24284 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:37.124467 24284 ts_tablet_manager.cc:1367] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:00:37.124898 24284 raft_consensus.cc:348] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.124992 24284 raft_consensus.cc:374] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.125036 24284 raft_consensus.cc:729] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.125147 24284 consensus_queue.cc:260] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.125412 24284 ts_tablet_manager.cc:1398] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.301826 24269 raft_consensus.cc:482] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.301978 24269 raft_consensus.cc:504] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.302384 24269 leader_election.cc:288] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:37.305742 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e8b697e8a4bf8a8b2ac5f9e0022a9" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:37.305931 23470 raft_consensus.cc:2454] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 0.
I20240502 14:00:37.306227 23536 leader_election.cc:302] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:37.306439 24269 raft_consensus.cc:2781] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.306499 24269 raft_consensus.cc:482] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.306526 24269 raft_consensus.cc:3037] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.307272 24269 raft_consensus.cc:504] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.307495 24269 leader_election.cc:288] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:37.307859 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e15e8b697e8a4bf8a8b2ac5f9e0022a9" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:37.307963 23470 raft_consensus.cc:3037] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.308821 23470 raft_consensus.cc:2454] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 1.
I20240502 14:00:37.309098 23536 leader_election.cc:302] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:37.309258 24269 raft_consensus.cc:2781] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.309334 24269 raft_consensus.cc:686] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Becoming Leader. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Running, Role: LEADER
I20240502 14:00:37.309464 24269 consensus_queue.cc:237] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.310787 23196 catalog_manager.cc:5484] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3). New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } }
I20240502 14:00:37.448976 24218 raft_consensus.cc:482] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.449106 24218 raft_consensus.cc:504] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.450007 24218 leader_election.cc:288] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:37.453737 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043771f4c844e86a8d7111f19c45389" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:37.453872 23601 raft_consensus.cc:2454] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:37.454272 23272 leader_election.cc:302] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:37.454488 24218 raft_consensus.cc:2781] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.454541 24218 raft_consensus.cc:482] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.454566 24218 raft_consensus.cc:3037] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.455410 24218 raft_consensus.cc:504] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.455667 24218 leader_election.cc:288] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:37.456005 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043771f4c844e86a8d7111f19c45389" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:37.456086 23601 raft_consensus.cc:3037] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.456771 23601 raft_consensus.cc:2454] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:37.457018 23272 leader_election.cc:302] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:37.457187 24218 raft_consensus.cc:2781] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.457369 24218 raft_consensus.cc:686] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:37.457623 24218 consensus_queue.cc:237] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.459391 23196 catalog_manager.cc:5484] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
I20240502 14:00:37.492331 24218 raft_consensus.cc:482] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.492419 24218 raft_consensus.cc:504] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.492707 24218 leader_election.cc:288] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:37.495442 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:37.495549 23470 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:37.495848 23274 leader_election.cc:302] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:37.495991 24218 raft_consensus.cc:2781] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.496043 24218 raft_consensus.cc:482] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.496074 24218 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.496829 24218 raft_consensus.cc:504] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.497020 24218 leader_election.cc:288] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:37.497364 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:37.497457 23470 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.498107 23470 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:37.498384 23274 leader_election.cc:302] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:37.498545 24218 raft_consensus.cc:2781] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.498634 24218 raft_consensus.cc:686] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:37.498757 24218 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:37.500088 23196 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } }
W20240502 14:00:37.660809 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:37.661005 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:37.663538 24297 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:00:37.727075 23470 raft_consensus.cc:1261] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.727658 24269 consensus_queue.cc:1035] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.729950 23469 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.729949 23601 raft_consensus.cc:1261] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:37.731197 24294 consensus_queue.cc:1035] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.731575 24294 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:37.739084 24329 mvcc.cc:205] Tried to move back new op lower bound from 7023240960936652800 to 7023240959828037632. Current Snapshot: MvccSnapshot[applied={T|T < 7023240960936652800}]
INSERT report
rows total: 2000
time total: 67.7659 ms
time per row: 0.033883 ms
I20240502 14:00:37.766743 23195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:37.771472 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 613713c348d64007b0cf596bca97713f (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.771519 23318 tablet_service.cc:1467] Processing CreateTablet for tablet 613713c348d64007b0cf596bca97713f (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.771626 23580 tablet_service.cc:1467] Processing CreateTablet for tablet d437d8593be743ef8a883ae31d8d5b24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.771802 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613713c348d64007b0cf596bca97713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.771822 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613713c348d64007b0cf596bca97713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.771865 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437d8593be743ef8a883ae31d8d5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.771951 23581 tablet_service.cc:1467] Processing CreateTablet for tablet 78f1ce57b0794e2d9f82d9948ffa9568 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.772161 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f1ce57b0794e2d9f82d9948ffa9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.774113 24349 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:37.774333 23318 tablet_service.cc:1467] Processing CreateTablet for tablet d437d8593be743ef8a883ae31d8d5b24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.774406 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 78f1ce57b0794e2d9f82d9948ffa9568 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:37.774535 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437d8593be743ef8a883ae31d8d5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.774583 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f1ce57b0794e2d9f82d9948ffa9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:37.774936 24349 tablet_bootstrap.cc:654] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.774969 24347 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:37.775966 24349 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:37.776032 24347 tablet_bootstrap.cc:654] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.776046 24349 ts_tablet_manager.cc:1367] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:37.776494 24349 raft_consensus.cc:348] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.776580 24349 raft_consensus.cc:374] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.776613 24349 raft_consensus.cc:729] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.776731 24349 consensus_queue.cc:260] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.776780 24348 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:37.776990 24349 ts_tablet_manager.cc:1398] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.777140 24349 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:37.777135 24347 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:37.777195 24347 ts_tablet_manager.cc:1367] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:37.777570 24347 raft_consensus.cc:348] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.777649 24347 raft_consensus.cc:374] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.777659 24348 tablet_bootstrap.cc:654] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.777675 24347 raft_consensus.cc:729] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.777796 24347 consensus_queue.cc:260] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.777878 24349 tablet_bootstrap.cc:654] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.778057 24347 ts_tablet_manager.cc:1398] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.778187 24347 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:37.778548 24348 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:37.778625 24348 ts_tablet_manager.cc:1367] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:37.779059 24347 tablet_bootstrap.cc:654] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.779013 24348 raft_consensus.cc:348] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.779095 24348 raft_consensus.cc:374] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.779134 24348 raft_consensus.cc:729] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.779237 24348 consensus_queue.cc:260] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.779480 24348 ts_tablet_manager.cc:1398] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.779774 24348 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:37.780519 24348 tablet_bootstrap.cc:654] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:37.780875 24349 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:37.780948 24349 ts_tablet_manager.cc:1367] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:37.781322 24349 raft_consensus.cc:348] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.781404 24349 raft_consensus.cc:374] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.781435 24349 raft_consensus.cc:729] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.781529 24349 consensus_queue.cc:260] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.781733 24349 ts_tablet_manager.cc:1398] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:37.784832 24348 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:37.784914 24348 ts_tablet_manager.cc:1367] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:00:37.785300 24348 raft_consensus.cc:348] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.785392 24348 raft_consensus.cc:374] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.785434 24348 raft_consensus.cc:729] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.785547 24348 consensus_queue.cc:260] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.785784 24348 ts_tablet_manager.cc:1398] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.786419 24347 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:37.786485 24347 ts_tablet_manager.cc:1367] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:00:37.786811 24347 raft_consensus.cc:348] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.786875 24347 raft_consensus.cc:374] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:37.786897 24347 raft_consensus.cc:729] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:37.786991 24347 consensus_queue.cc:260] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:37.787184 24347 ts_tablet_manager.cc:1398] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:37.898742 24334 raft_consensus.cc:482] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:37.898857 24334 raft_consensus.cc:504] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.899137 24334 leader_election.cc:288] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:37.899549 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613713c348d64007b0cf596bca97713f" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8" is_pre_election: true
I20240502 14:00:37.899693 23338 raft_consensus.cc:2454] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 0.
I20240502 14:00:37.899977 23404 leader_election.cc:302] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:37.900131 24334 raft_consensus.cc:2781] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:37.900180 24334 raft_consensus.cc:482] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:37.900204 24334 raft_consensus.cc:3037] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.901106 24334 raft_consensus.cc:504] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.901306 24334 leader_election.cc:288] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:37.901640 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613713c348d64007b0cf596bca97713f" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:37.901757 23338 raft_consensus.cc:3037] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:37.902468 23338 raft_consensus.cc:2454] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 1.
I20240502 14:00:37.902734 23404 leader_election.cc:302] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:37.902909 24334 raft_consensus.cc:2781] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:37.902977 24334 raft_consensus.cc:686] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Becoming Leader. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:37.903105 24334 consensus_queue.cc:237] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:37.904457 23196 catalog_manager.cc:5484] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49daee87c26c4b989eaec212705a11f9 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
I20240502 14:00:38.008725 24269 raft_consensus.cc:482] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.008834 24269 raft_consensus.cc:504] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.009148 24269 leader_election.cc:288] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:38.009526 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f1ce57b0794e2d9f82d9948ffa9568" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:38.009649 23469 raft_consensus.cc:2454] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 0.
I20240502 14:00:38.009879 23536 leader_election.cc:302] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.010039 24269 raft_consensus.cc:2781] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.010107 24269 raft_consensus.cc:482] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.010131 24269 raft_consensus.cc:3037] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.010852 24269 raft_consensus.cc:504] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.011051 24269 leader_election.cc:288] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:38.011390 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f1ce57b0794e2d9f82d9948ffa9568" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:38.011477 23469 raft_consensus.cc:3037] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.012154 23469 raft_consensus.cc:2454] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 1.
I20240502 14:00:38.012423 23536 leader_election.cc:302] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.012586 24269 raft_consensus.cc:2781] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.012686 24269 raft_consensus.cc:686] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Becoming Leader. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Running, Role: LEADER
I20240502 14:00:38.012832 24269 consensus_queue.cc:237] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.014199 23196 catalog_manager.cc:5484] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3). New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } }
I20240502 14:00:38.067555 24218 raft_consensus.cc:482] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.067668 24218 raft_consensus.cc:504] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.068017 24218 leader_election.cc:288] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.068511 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437d8593be743ef8a883ae31d8d5b24" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:38.068650 23601 raft_consensus.cc:2454] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:38.068986 23272 leader_election.cc:302] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.069172 24218 raft_consensus.cc:2781] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.069231 24218 raft_consensus.cc:482] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.069255 24218 raft_consensus.cc:3037] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.069931 24218 raft_consensus.cc:504] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.070132 24218 leader_election.cc:288] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.070438 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d437d8593be743ef8a883ae31d8d5b24" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:38.070520 23601 raft_consensus.cc:3037] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.071205 23601 raft_consensus.cc:2454] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:38.071470 23272 leader_election.cc:302] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.071610 24218 raft_consensus.cc:2781] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.071712 24218 raft_consensus.cc:686] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:38.071870 24218 consensus_queue.cc:237] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.073328 23196 catalog_manager.cc:5484] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
W20240502 14:00:38.224889 24359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:38.225080 24359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:38.227782 24359 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:00:38.297516 23338 raft_consensus.cc:1261] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.298022 24334 consensus_queue.cc:1035] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.300365 23469 raft_consensus.cc:1261] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.300750 24269 consensus_queue.cc:1035] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.306577 23601 raft_consensus.cc:1261] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.306996 24218 consensus_queue.cc:1035] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 77.2884 ms
time per row: 0.0386442 ms
I20240502 14:00:38.340734 23195 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:38.345194 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 7184d52ecc4a4922bbbd4541eab35d74 (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.345328 23318 tablet_service.cc:1467] Processing CreateTablet for tablet 7184d52ecc4a4922bbbd4541eab35d74 (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.345402 23580 tablet_service.cc:1467] Processing CreateTablet for tablet 6f625e35721741be82000754b5b1cf2c (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.345500 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184d52ecc4a4922bbbd4541eab35d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.345577 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184d52ecc4a4922bbbd4541eab35d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.345600 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f625e35721741be82000754b5b1cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.345636 23581 tablet_service.cc:1467] Processing CreateTablet for tablet 5a11bbc1c922463e9c73acc2966972ac (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.345819 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a11bbc1c922463e9c73acc2966972ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.347595 23319 tablet_service.cc:1467] Processing CreateTablet for tablet 6f625e35721741be82000754b5b1cf2c (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.347800 23319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f625e35721741be82000754b5b1cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.348037 24407 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:38.348098 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 5a11bbc1c922463e9c73acc2966972ac (DEFAULT_TABLE table=rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:38.348325 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a11bbc1c922463e9c73acc2966972ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:38.348841 24407 tablet_bootstrap.cc:654] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.349437 24408 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:38.349893 24407 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:38.349964 24409 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:38.350052 24407 ts_tablet_manager.cc:1367] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:38.350211 24408 tablet_bootstrap.cc:654] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.350553 24407 raft_consensus.cc:348] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.350718 24407 raft_consensus.cc:374] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.350744 24409 tablet_bootstrap.cc:654] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.350805 24407 raft_consensus.cc:729] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.350983 24407 consensus_queue.cc:260] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.351070 24408 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:38.351146 24408 ts_tablet_manager.cc:1367] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:38.351471 24408 raft_consensus.cc:348] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.351559 24408 raft_consensus.cc:374] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.351588 24408 raft_consensus.cc:729] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.351646 24407 ts_tablet_manager.cc:1398] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.351665 24409 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:38.351711 24408 consensus_queue.cc:260] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.351758 24409 ts_tablet_manager.cc:1367] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:38.351805 24407 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:38.351940 24408 ts_tablet_manager.cc:1398] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.352072 24408 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:38.352130 24409 raft_consensus.cc:348] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.352262 24409 raft_consensus.cc:374] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.352319 24409 raft_consensus.cc:729] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.352470 24409 consensus_queue.cc:260] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.352824 24409 ts_tablet_manager.cc:1398] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:38.352983 24409 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:38.352598 24407 tablet_bootstrap.cc:654] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.353726 24409 tablet_bootstrap.cc:654] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.353951 24407 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:38.354033 24407 ts_tablet_manager.cc:1367] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:38.354421 24407 raft_consensus.cc:348] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.354512 24407 raft_consensus.cc:374] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.354542 24407 raft_consensus.cc:729] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.354622 24407 consensus_queue.cc:260] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.354809 24408 tablet_bootstrap.cc:654] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:38.354856 24407 ts_tablet_manager.cc:1398] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.355720 24408 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:38.355787 24408 ts_tablet_manager.cc:1367] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:00:38.356163 24408 raft_consensus.cc:348] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.356226 24409 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:38.356247 24408 raft_consensus.cc:374] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.356310 24409 ts_tablet_manager.cc:1367] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:00:38.356314 24408 raft_consensus.cc:729] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.356426 24408 consensus_queue.cc:260] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.356671 24408 ts_tablet_manager.cc:1398] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.356707 24409 raft_consensus.cc:348] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.356804 24409 raft_consensus.cc:374] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:38.356838 24409 raft_consensus.cc:729] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:38.356935 24409 consensus_queue.cc:260] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.357383 24409 ts_tablet_manager.cc:1398] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:38.430265 24218 raft_consensus.cc:482] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.430377 24218 raft_consensus.cc:504] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.430675 24218 leader_election.cc:288] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:38.431047 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:38.431181 23469 raft_consensus.cc:2454] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:38.431407 23274 leader_election.cc:302] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:38.431562 24218 raft_consensus.cc:2781] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.431615 24218 raft_consensus.cc:482] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.431643 24218 raft_consensus.cc:3037] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.432485 24218 raft_consensus.cc:504] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.432708 24218 leader_election.cc:288] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:38.433102 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:38.433194 23469 raft_consensus.cc:3037] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.433840 23469 raft_consensus.cc:2454] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:38.434072 23274 leader_election.cc:302] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:38.434227 24218 raft_consensus.cc:2781] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.434298 24218 raft_consensus.cc:686] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:38.434425 24218 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } }
I20240502 14:00:38.435752 23196 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } }
I20240502 14:00:38.590425 24218 raft_consensus.cc:482] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.590551 24218 raft_consensus.cc:504] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.590893 24218 leader_election.cc:288] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.591325 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f625e35721741be82000754b5b1cf2c" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:38.591459 23601 raft_consensus.cc:2454] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:38.591745 23272 leader_election.cc:302] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.591917 24218 raft_consensus.cc:2781] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.591971 24218 raft_consensus.cc:482] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.592012 24218 raft_consensus.cc:3037] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.592873 24218 raft_consensus.cc:504] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.593109 24218 leader_election.cc:288] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.593432 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f625e35721741be82000754b5b1cf2c" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:38.593526 23601 raft_consensus.cc:3037] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.594252 23601 raft_consensus.cc:2454] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:38.594497 23272 leader_election.cc:302] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.594648 24218 raft_consensus.cc:2781] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.594745 24218 raft_consensus.cc:686] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:38.594872 24218 consensus_queue.cc:237] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.596477 23196 catalog_manager.cc:5484] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } }
I20240502 14:00:38.751766 24334 raft_consensus.cc:482] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:38.751865 24334 raft_consensus.cc:504] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.752111 24334 leader_election.cc:288] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.752501 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a11bbc1c922463e9c73acc2966972ac" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:38.752629 23601 raft_consensus.cc:2454] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 0.
I20240502 14:00:38.752928 23403 leader_election.cc:302] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.753089 24334 raft_consensus.cc:2781] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:38.753165 24334 raft_consensus.cc:482] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:38.753213 24334 raft_consensus.cc:3037] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.753929 24334 raft_consensus.cc:504] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.754170 24334 leader_election.cc:288] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:38.754463 23601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a11bbc1c922463e9c73acc2966972ac" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:38.754557 23601 raft_consensus.cc:3037] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:38.755232 23601 raft_consensus.cc:2454] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 1.
I20240502 14:00:38.755520 23403 leader_election.cc:302] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:38.755662 24334 raft_consensus.cc:2781] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:38.755738 24334 raft_consensus.cc:686] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Becoming Leader. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:38.755856 24334 consensus_queue.cc:237] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:38.757288 23196 catalog_manager.cc:5484] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49daee87c26c4b989eaec212705a11f9 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } }
I20240502 14:00:38.879925 24218 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:38.894114 24419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:38.894292 24419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:38.897241 24419 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:00:38.971073 23600 raft_consensus.cc:1261] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.971073 23601 raft_consensus.cc:1261] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:38.971920 24334 consensus_queue.cc:1035] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:38.971972 24294 consensus_queue.cc:1035] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 83.5359 ms
time per row: 0.0417679 ms
I20240502 14:00:39.016479 23196 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45914:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:00:39.021157 23319 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd0482496c5456c9b491d0d92eecd03 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.021330 23318 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4503e308724cddb24e1dd16db817ff (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.021371 23449 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd0482496c5456c9b491d0d92eecd03 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.021461 23319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd0482496c5456c9b491d0d92eecd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.021643 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd0482496c5456c9b491d0d92eecd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.021842 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4503e308724cddb24e1dd16db817ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.022024 23580 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4503e308724cddb24e1dd16db817ff (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.022248 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4503e308724cddb24e1dd16db817ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.023690 24471 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:39.023753 23581 tablet_service.cc:1467] Processing CreateTablet for tablet de7005115c824c009aba7de17c8b68ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.023950 23581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7005115c824c009aba7de17c8b68ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.024392 23450 tablet_service.cc:1467] Processing CreateTablet for tablet de7005115c824c009aba7de17c8b68ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:00:39.024530 24471 tablet_bootstrap.cc:654] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.024601 23450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7005115c824c009aba7de17c8b68ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:39.025477 24470 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:39.025569 24471 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:39.025681 24471 ts_tablet_manager.cc:1367] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.026185 24472 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:39.026151 24471 raft_consensus.cc:348] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.026252 24471 raft_consensus.cc:374] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.026288 24471 raft_consensus.cc:729] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.026396 24471 consensus_queue.cc:260] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.026654 24471 ts_tablet_manager.cc:1398] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.026827 24471 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Bootstrap starting.
I20240502 14:00:39.026943 24472 tablet_bootstrap.cc:654] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.027632 24470 tablet_bootstrap.cc:654] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.027670 24471 tablet_bootstrap.cc:654] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.028759 24470 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:39.028832 24470 ts_tablet_manager.cc:1367] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:39.028879 24472 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:39.028966 24472 ts_tablet_manager.cc:1367] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:39.029294 24470 raft_consensus.cc:348] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.029373 24470 raft_consensus.cc:374] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.029403 24470 raft_consensus.cc:729] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.029397 24472 raft_consensus.cc:348] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.029489 24472 raft_consensus.cc:374] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.029515 24472 raft_consensus.cc:729] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.029513 24470 consensus_queue.cc:260] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.029740 24470 ts_tablet_manager.cc:1398] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.029886 24470 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Bootstrap starting.
I20240502 14:00:39.029755 24472 consensus_queue.cc:260] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.030378 24472 ts_tablet_manager.cc:1398] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.030491 24472 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: Bootstrap starting.
I20240502 14:00:39.030587 24470 tablet_bootstrap.cc:654] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.031095 24472 tablet_bootstrap.cc:654] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: Neither blocks nor log segments found. Creating new log.
I20240502 14:00:39.031641 24470 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: No bootstrap required, opened a new log
I20240502 14:00:39.031734 24470 ts_tablet_manager.cc:1367] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.032181 24472 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: No bootstrap required, opened a new log
I20240502 14:00:39.032191 24470 raft_consensus.cc:348] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.032298 24472 ts_tablet_manager.cc:1367] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:39.032292 24470 raft_consensus.cc:374] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.032354 24470 raft_consensus.cc:729] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.032490 24470 consensus_queue.cc:260] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.032795 24470 ts_tablet_manager.cc:1398] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.033174 24472 raft_consensus.cc:348] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.033272 24472 raft_consensus.cc:374] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.033300 24471 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: No bootstrap required, opened a new log
I20240502 14:00:39.033303 24472 raft_consensus.cc:729] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.033370 24471 ts_tablet_manager.cc:1367] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:00:39.033393 24472 consensus_queue.cc:260] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.033686 24472 ts_tablet_manager.cc:1398] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:39.033747 24471 raft_consensus.cc:348] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.033864 24471 raft_consensus.cc:374] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:00:39.033907 24471 raft_consensus.cc:729] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.034039 24471 consensus_queue.cc:260] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.034265 24471 ts_tablet_manager.cc:1398] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:39.041853 24452 raft_consensus.cc:482] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.041936 24452 raft_consensus.cc:504] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.042229 24452 leader_election.cc:288] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:39.042593 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7005115c824c009aba7de17c8b68ee" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:39.042716 23469 raft_consensus.cc:2454] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 0.
I20240502 14:00:39.042977 23536 leader_election.cc:302] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:39.043128 24452 raft_consensus.cc:2781] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.043187 24452 raft_consensus.cc:482] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.043228 24452 raft_consensus.cc:3037] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.043926 24452 raft_consensus.cc:504] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.044178 24452 leader_election.cc:288] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:39.044499 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de7005115c824c009aba7de17c8b68ee" candidate_uuid: "968c2f6eed3842dfa034660fa0a2117b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:39.044589 23469 raft_consensus.cc:3037] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.045310 23469 raft_consensus.cc:2454] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c2f6eed3842dfa034660fa0a2117b in term 1.
I20240502 14:00:39.045598 23536 leader_election.cc:302] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:39.045753 24452 raft_consensus.cc:2781] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.045845 24452 raft_consensus.cc:686] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Becoming Leader. State: Replica: 968c2f6eed3842dfa034660fa0a2117b, State: Running, Role: LEADER
I20240502 14:00:39.045960 24452 consensus_queue.cc:237] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.047403 23192 catalog_manager.cc:5484] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3). New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } }
I20240502 14:00:39.323297 24294 raft_consensus.cc:482] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.323410 24294 raft_consensus.cc:504] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.323701 24294 leader_election.cc:288] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:39.324080 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9" is_pre_election: true
I20240502 14:00:39.324204 23469 raft_consensus.cc:2454] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:39.324470 23274 leader_election.cc:302] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:39.324626 24294 raft_consensus.cc:2781] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.324676 24294 raft_consensus.cc:482] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.324700 24294 raft_consensus.cc:3037] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.325397 24294 raft_consensus.cc:504] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.325598 24294 leader_election.cc:288] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:39.325933 23469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:39.326021 23469 raft_consensus.cc:3037] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.326774 23469 raft_consensus.cc:2454] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:39.327016 23274 leader_election.cc:302] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:39.327159 24294 raft_consensus.cc:2781] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.327239 24294 raft_consensus.cc:686] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:39.327373 24294 consensus_queue.cc:237] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } }
I20240502 14:00:39.328917 23192 catalog_manager.cc:5484] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } }
I20240502 14:00:39.377604 24294 raft_consensus.cc:482] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:39.377719 24294 raft_consensus.cc:504] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.378043 24294 leader_election.cc:288] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:39.378448 23600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b" is_pre_election: true
I20240502 14:00:39.378571 23600 raft_consensus.cc:2454] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 0.
I20240502 14:00:39.378870 23272 leader_election.cc:302] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:39.379036 24294 raft_consensus.cc:2781] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:00:39.379091 24294 raft_consensus.cc:482] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:39.379129 24294 raft_consensus.cc:3037] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.379913 24294 raft_consensus.cc:504] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.380146 24294 leader_election.cc:288] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Requested vote from peers 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:39.380511 23600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff" candidate_uuid: "6395860ed21149cab200c8364faf6fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c2f6eed3842dfa034660fa0a2117b"
I20240502 14:00:39.380613 23600 raft_consensus.cc:3037] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:00:39.381343 23600 raft_consensus.cc:2454] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6395860ed21149cab200c8364faf6fa8 in term 1.
I20240502 14:00:39.381626 23272 leader_election.cc:302] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 968c2f6eed3842dfa034660fa0a2117b; no voters:
I20240502 14:00:39.381805 24294 raft_consensus.cc:2781] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:00:39.381911 24294 raft_consensus.cc:686] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:39.382061 24294 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } }
I20240502 14:00:39.383495 23192 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6395860ed21149cab200c8364faf6fa8 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } }
I20240502 14:00:39.460041 24452 consensus_queue.cc:1035] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:00:39.571710 24483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:39.571908 24483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:39.574590 24483 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:00:39.650869 23469 raft_consensus.cc:1261] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:39.651348 24294 consensus_queue.cc:1035] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:39.654821 23600 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:00:39.655146 24294 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 73.2576 ms
time per row: 0.0366288 ms
I20240502 14:00:39.684314 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.4:41841
--local_ip_for_outbound_sockets=127.16.217.4
--tserver_master_addrs=127.16.217.62:32783
--webserver_port=35743
--webserver_interface=127.16.217.4
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:00:39.777364 24525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:39.777523 24525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:39.777558 24525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:39.780385 24525 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:39.780434 24525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:39.780500 24525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.4
I20240502 14:00:39.784686 24525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.4:41841
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.217.4
--webserver_port=35743
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.24525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.4
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:39.785490 24525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:39.786569 24525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:39.795598 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:39.795707 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:39.796079 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:39.796773 24525 server_base.cc:1034] running on GCE node
I20240502 14:00:39.797133 24525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:39.797621 24525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:39.798770 24525 hybrid_clock.cc:648] HybridClock initialized: now 1714658439798756 us; error 28 us; skew 500 ppm
I20240502 14:00:39.800494 24525 webserver.cc:460] Webserver started at http://127.16.217.4:35743/ using document root <none> and password file <none>
I20240502 14:00:39.801021 24525 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:39.801098 24525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:39.804128 24525 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:39.805827 24539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:39.806672 24525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 14:00:39.806757 24525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "916fb151c5274f2c8caed395dcdf412b"
format_stamp: "Formatted at 2024-05-02 14:00:31 on dist-test-slave-7msf"
I20240502 14:00:39.807068 24525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:39.817402 24525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:39.817860 24525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:39.817965 24525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:39.818349 24525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:39.819216 24546 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 14:00:39.821918 24525 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:00:39.821962 24525 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240502 14:00:39.822001 24525 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:00:39.823256 24525 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:00:39.823294 24525 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240502 14:00:39.823521 24546 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:39.832412 24525 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.4:41841
I20240502 14:00:39.832435 24653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.4:41841 every 8 connection(s)
I20240502 14:00:39.833334 24525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:00:39.835681 24546 log.cc:824] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:39.837625 24546 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:39.837966 24546 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap complete.
I20240502 14:00:39.839028 24546 ts_tablet_manager.cc:1367] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20240502 14:00:39.839110 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 24525
I20240502 14:00:39.839457 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:43149
--local_ip_for_outbound_sockets=127.16.217.5
--tserver_master_addrs=127.16.217.62:32783
--webserver_port=44519
--webserver_interface=127.16.217.5
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:39.841446 24654 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:39.841645 24654 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:39.842046 24654 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:39.842022 24546 raft_consensus.cc:348] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:39.842434 24546 raft_consensus.cc:729] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:39.842809 24546 consensus_queue.cc:260] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:39.843533 24546 ts_tablet_manager.cc:1398] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:00:39.844487 23190 ts_manager.cc:194] Re-registered known tserver with Master: 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841)
I20240502 14:00:39.845131 23190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.4:37271
I20240502 14:00:39.847131 24654 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
W20240502 14:00:39.940027 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:39.940230 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:39.940306 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:39.943048 24658 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:39.943110 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:39.943207 24658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 14:00:39.946273 24658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:43149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=44519
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.24658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:39.946991 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:39.948004 24658 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:39.955417 24664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:39.955474 24665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:39.956030 24667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:39.956749 24658 server_base.cc:1034] running on GCE node
I20240502 14:00:39.957049 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:39.957484 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:39.958627 24658 hybrid_clock.cc:648] HybridClock initialized: now 1714658439958607 us; error 32 us; skew 500 ppm
I20240502 14:00:39.960304 24658 webserver.cc:460] Webserver started at http://127.16.217.5:44519/ using document root <none> and password file <none>
I20240502 14:00:39.960815 24658 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:39.960886 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:39.963747 24658 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:00:39.965402 24673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:39.966274 24658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:39.966361 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "17a03815fc864e4db7a68c32c6e42616"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:39.966745 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:39.976498 24658 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:39.976943 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:39.977077 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:39.977501 24658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:39.978195 24658 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:39.978236 24658 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:39.978268 24658 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:39.978296 24658 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:39.986047 24658 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:43149
I20240502 14:00:39.986086 24787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:43149 every 8 connection(s)
I20240502 14:00:39.986851 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:00:39.987604 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 24658
I20240502 14:00:39.988072 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:42367
--local_ip_for_outbound_sockets=127.16.217.6
--tserver_master_addrs=127.16.217.62:32783
--webserver_port=41419
--webserver_interface=127.16.217.6
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:39.995404 24788 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:39.995640 24788 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:39.996162 24788 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:39.997290 23190 ts_manager.cc:194] Re-registered known tserver with Master: 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149)
I20240502 14:00:39.997807 23190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:48699
W20240502 14:00:40.084193 24655 tablet.cc:2369] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:00:40.085947 24791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:40.086099 24791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:40.086133 24791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:40.088613 24791 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:40.088660 24791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:40.088712 24791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 14:00:40.091789 24791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:42367
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=41419
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.24791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:40.092535 24791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:40.093537 24791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:40.101105 24800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:40.101565 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:40.101727 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:40.102424 24791 server_base.cc:1034] running on GCE node
I20240502 14:00:40.102743 24791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:40.103260 24791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:40.104411 24791 hybrid_clock.cc:648] HybridClock initialized: now 1714658440104393 us; error 36 us; skew 500 ppm
I20240502 14:00:40.105971 24791 webserver.cc:460] Webserver started at http://127.16.217.6:41419/ using document root <none> and password file <none>
I20240502 14:00:40.106457 24791 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:40.106530 24791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:40.109663 24791 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:40.111220 24806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:40.112104 24791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:00:40.112196 24791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "66c6a4ade9e84186beea567c21f80eb8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:40.112586 24791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:40.128767 24791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:40.129376 24791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:40.129511 24791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:40.129956 24791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:40.130686 24791 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:00:40.130728 24791 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:40.130775 24791 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:00:40.130803 24791 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:40.138398 24791 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:42367
I20240502 14:00:40.138446 24920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:42367 every 8 connection(s)
I20240502 14:00:40.139245 24791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:00:40.143541 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 24791
I20240502 14:00:40.143899 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:32823
--local_ip_for_outbound_sockets=127.16.217.7
--tserver_master_addrs=127.16.217.62:32783
--webserver_port=39889
--webserver_interface=127.16.217.7
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:00:40.147449 24921 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:40.147622 24921 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:40.148018 24921 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:40.148944 23190 ts_manager.cc:194] Re-registered known tserver with Master: 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367)
I20240502 14:00:40.149435 23190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:37547
W20240502 14:00:40.240746 24924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:00:40.240907 24924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:00:40.240967 24924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:00:40.243669 24924 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:00:40.243723 24924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:00:40.243805 24924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 14:00:40.246743 24924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:32823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=39889
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.24924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:00:40.247437 24924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:00:40.248529 24924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:00:40.255549 24932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:40.256047 24929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:00:40.256588 24930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:00:40.257323 24924 server_base.cc:1034] running on GCE node
I20240502 14:00:40.257608 24924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:00:40.258090 24924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:00:40.259182 24924 hybrid_clock.cc:648] HybridClock initialized: now 1714658440259158 us; error 44 us; skew 500 ppm
I20240502 14:00:40.260787 24924 webserver.cc:460] Webserver started at http://127.16.217.7:39889/ using document root <none> and password file <none>
I20240502 14:00:40.261236 24924 fs_manager.cc:359] Metadata directory not provided
I20240502 14:00:40.261301 24924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:00:40.264173 24924 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:00:40.265766 24938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:00:40.266577 24924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:40.266674 24924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a3101d9789354f17b9af4da59878eec8"
format_stamp: "Formatted at 2024-05-02 14:00:32 on dist-test-slave-7msf"
I20240502 14:00:40.266992 24924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:00:40.275578 24924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:00:40.276118 24924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:00:40.276304 24924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:00:40.276820 24924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:00:40.277859 24945 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 14:00:40.280673 24924 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:00:40.280742 24924 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240502 14:00:40.280787 24924 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:00:40.282341 24924 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:00:40.282388 24924 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 14:00:40.282624 24945 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Bootstrap starting.
I20240502 14:00:40.291795 24924 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:32823
I20240502 14:00:40.291826 25055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:32823 every 8 connection(s)
I20240502 14:00:40.292819 24924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:00:40.295806 24945 log.cc:824] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:40.297081 24945 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:40.297334 24945 tablet_bootstrap.cc:492] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Bootstrap complete.
I20240502 14:00:40.298449 24945 ts_tablet_manager.cc:1367] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 14:00:40.299474 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 24924
I20240502 14:00:40.299624 17252 test_util.cc:268] Using random seed: 121554023
I20240502 14:00:40.301501 25056 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:00:40.301702 25056 heartbeater.cc:464] Registering TS with master...
I20240502 14:00:40.301443 24945 raft_consensus.cc:348] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:40.301817 24945 raft_consensus.cc:729] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Initialized, Role: FOLLOWER
I20240502 14:00:40.302201 25056 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:00:40.302248 24945 consensus_queue.cc:260] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:40.303076 24945 ts_tablet_manager.cc:1398] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:00:40.303915 23190 ts_manager.cc:194] Re-registered known tserver with Master: a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:40.304679 23190 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:51411
I20240502 14:00:40.306563 25056 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:40.310385 17252 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 14:00:40.310572 17252 test_util.cc:268] Using random seed: 121564971
I20240502 14:00:40.318017 17252 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 14:00:40.318169 17252 test_util.cc:268] Using random seed: 121572567
I20240502 14:00:40.328855 17252 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 14:00:40.328998 17252 test_util.cc:268] Using random seed: 121583396
I20240502 14:00:40.338891 17252 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 14:00:40.339010 17252 test_util.cc:268] Using random seed: 121593408
I20240502 14:00:40.351708 17252 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240502 14:00:40.543676 25057 tablet.cc:2369] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:00:40.734020 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:40.747651 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:40.762904 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:40.766359 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.777841 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.780267 24588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:40.793550 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:41.000000 24788 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:41.052632 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:41.058365 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:41.073714 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:41.073735 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:41.111634 24588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:41.136662 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:41.137279 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:41.150965 24921 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:00:41.230834 23339 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:41.233357 23468 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 14:00:41.235069 25093 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 14:00:41.236727 25093 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:41.238018 23467 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:41.240831 23196 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index -1 to 37, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
W20240502 14:00:41.254052 23273 consensus_peers.cc:485] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 -> Peer 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367): Couldn't send request to peer 66c6a4ade9e84186beea567c21f80eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7184d52ecc4a4922bbbd4541eab35d74. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.327482 23338 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:41.330586 23599 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240502 14:00:41.332072 25093 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
W20240502 14:00:41.333503 23275 consensus_peers.cc:485] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 -> Peer 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149): Couldn't send request to peer 17a03815fc864e4db7a68c32c6e42616. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a4503e308724cddb24e1dd16db817ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.341630 24294 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.343228 23600 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.345049 23196 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 50, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.409703 23600 consensus_queue.cc:237] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:41.413621 25286 raft_consensus.cc:482] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:00:41.413923 25286 raft_consensus.cc:504] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:41.416633 25286 leader_election.cc:288] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:41.421913 25008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5ab6a504d4fceab781c331a40627d" candidate_uuid: "916fb151c5274f2c8caed395dcdf412b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3101d9789354f17b9af4da59878eec8" is_pre_election: true
I20240502 14:00:41.422235 25008 raft_consensus.cc:2454] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916fb151c5274f2c8caed395dcdf412b in term 1.
I20240502 14:00:41.422654 24542 leader_election.cc:302] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 916fb151c5274f2c8caed395dcdf412b, a3101d9789354f17b9af4da59878eec8; no voters:
I20240502 14:00:41.422863 25286 raft_consensus.cc:2781] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:00:41.422986 25286 raft_consensus.cc:482] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:00:41.423108 25286 raft_consensus.cc:3037] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:41.425370 25286 raft_consensus.cc:504] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:41.426007 25008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb5ab6a504d4fceab781c331a40627d" candidate_uuid: "916fb151c5274f2c8caed395dcdf412b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3101d9789354f17b9af4da59878eec8"
I20240502 14:00:41.426127 25008 raft_consensus.cc:3037] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:41.427978 25008 raft_consensus.cc:2454] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916fb151c5274f2c8caed395dcdf412b in term 2.
I20240502 14:00:41.428251 24542 leader_election.cc:302] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 916fb151c5274f2c8caed395dcdf412b, a3101d9789354f17b9af4da59878eec8; no voters:
I20240502 14:00:41.428653 25286 leader_election.cc:288] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [CANDIDATE]: Term 2 election: Requested vote from peers a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823)
I20240502 14:00:41.428556 25288 raft_consensus.cc:2781] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:41.429697 25288 raft_consensus.cc:686] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [term 2 LEADER]: Becoming Leader. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Running, Role: LEADER
I20240502 14:00:41.429945 25288 consensus_queue.cc:237] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:41.432646 23193 catalog_manager.cc:5484] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "916fb151c5274f2c8caed395dcdf412b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:41.438289 23467 raft_consensus.cc:1261] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20240502 14:00:41.438890 25086 consensus_queue.cc:1035] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20240502 14:00:41.441247 25137 raft_consensus.cc:2932] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index -1 to 34, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:41.445961 23193 catalog_manager.cc:5484] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index -1 to 34, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:41.446983 23467 raft_consensus.cc:2932] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index -1 to 34, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
W20240502 14:00:41.447149 23536 consensus_peers.cc:485] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b -> Peer a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823): Couldn't send request to peer a3101d9789354f17b9af4da59878eec8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de7005115c824c009aba7de17c8b68ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.558116 23468 consensus_queue.cc:237] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:41.563808 23601 raft_consensus.cc:1261] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 14:00:41.565208 25145 consensus_queue.cc:1035] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 14:00:41.570111 24334 raft_consensus.cc:2932] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:41.571064 23601 raft_consensus.cc:2932] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:41.577224 23196 catalog_manager.cc:5484] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index -1 to 51, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
W20240502 14:00:41.578532 23403 consensus_peers.cc:485] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 -> Peer 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841): Couldn't send request to peer 916fb151c5274f2c8caed395dcdf412b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b935eeb4b2344d78780f44fff524e9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.645267 23339 consensus_queue.cc:237] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
W20240502 14:00:41.646651 23273 consensus_peers.cc:485] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 -> Peer 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367): Couldn't send request to peer 66c6a4ade9e84186beea567c21f80eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dd0482496c5456c9b491d0d92eecd03. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.647560 23469 raft_consensus.cc:1261] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 14:00:41.647909 25093 consensus_queue.cc:1035] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20240502 14:00:41.651655 25142 raft_consensus.cc:2932] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:41.654966 23196 catalog_manager.cc:5484] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 68, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:41.656113 23468 raft_consensus.cc:2932] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:41.772712 23339 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
W20240502 14:00:41.775386 23275 consensus_peers.cc:485] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 -> Peer 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149): Couldn't send request to peer 17a03815fc864e4db7a68c32c6e42616. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbbf1e7447604baa9701d6330ff31a3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.775786 23469 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240502 14:00:41.777673 24294 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240502 14:00:41.785326 24294 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index -1 to 57, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.785702 23468 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index -1 to 57, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.788734 23196 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index -1 to 57, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:41.834839 25344 ts_tablet_manager.cc:902] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:41.857048 25344 tablet_copy_client.cc:308] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:41.865437 25008 raft_consensus.cc:1261] T 9eb5ab6a504d4fceab781c331a40627d P a3101d9789354f17b9af4da59878eec8 [term 2 FOLLOWER]: Refusing update from remote peer 916fb151c5274f2c8caed395dcdf412b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:00:41.867205 25288 consensus_queue.cc:1035] T 9eb5ab6a504d4fceab781c331a40627d P 916fb151c5274f2c8caed395dcdf412b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:00:41.880434 25361 ts_tablet_manager.cc:902] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:41.881238 25361 tablet_copy_client.cc:308] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:41.884624 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet 7184d52ecc4a4922bbbd4541eab35d74 from peer 66c6a4ade9e84186beea567c21f80eb8 ({username='slave'} at 127.16.217.6:40417)
I20240502 14:00:41.884764 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet 7184d52ecc4a4922bbbd4541eab35d74 from peer 66c6a4ade9e84186beea567c21f80eb8 at {username='slave'} at 127.16.217.6:40417: session id = 66c6a4ade9e84186beea567c21f80eb8-7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:00:41.885939 23359 tablet_copy_source_session.cc:205] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.890368 25344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184d52ecc4a4922bbbd4541eab35d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.898125 25372 ts_tablet_manager.cc:902] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Initiating tablet copy from peer 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:41.903554 23469 consensus_queue.cc:237] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:41.904736 23338 raft_consensus.cc:1261] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 14:00:41.905064 25332 consensus_queue.cc:1035] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 14:00:41.906858 25232 raft_consensus.cc:2932] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:41.908370 23338 raft_consensus.cc:2932] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
W20240502 14:00:41.908632 23405 consensus_peers.cc:485] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 -> Peer a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823): Couldn't send request to peer a3101d9789354f17b9af4da59878eec8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b29f51fa8f14e3bb8b859b1941e6fac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:41.908999 25372 tablet_copy_client.cc:308] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:33339
I20240502 14:00:41.911006 23193 catalog_manager.cc:5484] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 71, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:41.923089 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet 1a4503e308724cddb24e1dd16db817ff from peer 17a03815fc864e4db7a68c32c6e42616 ({username='slave'} at 127.16.217.5:58087)
I20240502 14:00:41.923199 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet 1a4503e308724cddb24e1dd16db817ff from peer 17a03815fc864e4db7a68c32c6e42616 at {username='slave'} at 127.16.217.5:58087: session id = 17a03815fc864e4db7a68c32c6e42616-1a4503e308724cddb24e1dd16db817ff
I20240502 14:00:41.924058 23359 tablet_copy_source_session.cc:205] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.927506 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4503e308724cddb24e1dd16db817ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.932726 23621 tablet_copy_service.cc:140] P 968c2f6eed3842dfa034660fa0a2117b: Received BeginTabletCopySession request for tablet de7005115c824c009aba7de17c8b68ee from peer a3101d9789354f17b9af4da59878eec8 ({username='slave'} at 127.16.217.7:49525)
I20240502 14:00:41.932868 23621 tablet_copy_service.cc:161] P 968c2f6eed3842dfa034660fa0a2117b: Beginning new tablet copy session on tablet de7005115c824c009aba7de17c8b68ee from peer a3101d9789354f17b9af4da59878eec8 at {username='slave'} at 127.16.217.7:49525: session id = a3101d9789354f17b9af4da59878eec8-de7005115c824c009aba7de17c8b68ee
I20240502 14:00:41.934041 23621 tablet_copy_source_session.cc:205] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:41.956670 25361 tablet_copy_client.cc:783] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.956990 25361 tablet_copy_client.cc:647] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.957960 25344 tablet_copy_client.cc:783] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.958156 25344 tablet_copy_client.cc:647] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.960048 25361 tablet_copy_client.cc:523] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.962041 25361 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Bootstrap starting.
I20240502 14:00:41.962807 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7005115c824c009aba7de17c8b68ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:41.968142 25344 tablet_copy_client.cc:523] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:41.974956 25372 tablet_copy_client.cc:783] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:41.975117 25372 tablet_copy_client.cc:647] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:41.975544 25344 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap starting.
I20240502 14:00:41.990504 25361 log.cc:824] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:42.000449 25344 log.cc:824] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:00:42.000593 25372 tablet_copy_client.cc:523] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.002092 25372 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Bootstrap starting.
I20240502 14:00:42.008755 23338 consensus_queue.cc:237] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.021216 23598 raft_consensus.cc:1261] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240502 14:00:42.024541 25142 consensus_queue.cc:1035] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240502 14:00:42.035115 25152 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.038161 23196 catalog_manager.cc:5484] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 60, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.040933 23598 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index -1 to 60, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.079811 25361 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.080188 25361 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Bootstrap complete.
I20240502 14:00:42.080550 25361 ts_tablet_manager.cc:1367] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Time spent bootstrapping tablet: real 0.119s user 0.075s sys 0.004s
I20240502 14:00:42.083113 25361 raft_consensus.cc:348] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:42.083360 25361 raft_consensus.cc:729] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a03815fc864e4db7a68c32c6e42616, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.083858 25361 consensus_queue.cc:260] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:42.084780 25361 ts_tablet_manager.cc:1398] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:00:42.089406 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 17a03815fc864e4db7a68c32c6e42616-1a4503e308724cddb24e1dd16db817ff received from {username='slave'} at 127.16.217.5:58087
I20240502 14:00:42.089491 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 17a03815fc864e4db7a68c32c6e42616-1a4503e308724cddb24e1dd16db817ff on tablet 1a4503e308724cddb24e1dd16db817ff with peer 17a03815fc864e4db7a68c32c6e42616
I20240502 14:00:42.090354 25404 ts_tablet_manager.cc:902] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Initiating tablet copy from peer 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:42.091615 25404 tablet_copy_client.cc:308] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:36885
I20240502 14:00:42.094408 25344 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:42.094830 25344 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap complete.
I20240502 14:00:42.095731 25344 ts_tablet_manager.cc:1367] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Time spent bootstrapping tablet: real 0.120s user 0.071s sys 0.005s
I20240502 14:00:42.098187 25344 raft_consensus.cc:348] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.099516 25344 raft_consensus.cc:729] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.099967 25344 consensus_queue.cc:260] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.100734 25344 ts_tablet_manager.cc:1398] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:00:42.102788 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 66c6a4ade9e84186beea567c21f80eb8-7184d52ecc4a4922bbbd4541eab35d74 received from {username='slave'} at 127.16.217.6:40417
I20240502 14:00:42.102869 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 66c6a4ade9e84186beea567c21f80eb8-7184d52ecc4a4922bbbd4541eab35d74 on tablet 7184d52ecc4a4922bbbd4541eab35d74 with peer 66c6a4ade9e84186beea567c21f80eb8
W20240502 14:00:42.124691 23272 consensus_peers.cc:485] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 -> Peer 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841): Couldn't send request to peer 916fb151c5274f2c8caed395dcdf412b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d437d8593be743ef8a883ae31d8d5b24. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.150650 23490 tablet_copy_service.cc:140] P 49daee87c26c4b989eaec212705a11f9: Received BeginTabletCopySession request for tablet 0b935eeb4b2344d78780f44fff524e9d from peer 916fb151c5274f2c8caed395dcdf412b ({username='slave'} at 127.16.217.4:41667)
I20240502 14:00:42.150785 23490 tablet_copy_service.cc:161] P 49daee87c26c4b989eaec212705a11f9: Beginning new tablet copy session on tablet 0b935eeb4b2344d78780f44fff524e9d from peer 916fb151c5274f2c8caed395dcdf412b at {username='slave'} at 127.16.217.4:41667: session id = 916fb151c5274f2c8caed395dcdf412b-0b935eeb4b2344d78780f44fff524e9d
I20240502 14:00:42.152890 23490 tablet_copy_source_session.cc:205] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.155007 25404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935eeb4b2344d78780f44fff524e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.157742 25404 tablet_copy_client.cc:783] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.157912 25404 tablet_copy_client.cc:647] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.161713 25404 tablet_copy_client.cc:523] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.168380 25372 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=55 ignored=0} inserts{seen=727 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.168668 25372 tablet_bootstrap.cc:492] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Bootstrap complete.
I20240502 14:00:42.168852 25372 ts_tablet_manager.cc:1367] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Time spent bootstrapping tablet: real 0.167s user 0.049s sys 0.012s
I20240502 14:00:42.169420 25372 raft_consensus.cc:348] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:42.169529 25372 raft_consensus.cc:729] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.169690 25372 consensus_queue.cc:260] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 55, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:42.169930 25372 ts_tablet_manager.cc:1398] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:00:42.171785 25404 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:42.171954 23621 tablet_copy_service.cc:342] P 968c2f6eed3842dfa034660fa0a2117b: Request end of tablet copy session a3101d9789354f17b9af4da59878eec8-de7005115c824c009aba7de17c8b68ee received from {username='slave'} at 127.16.217.7:49525
I20240502 14:00:42.172055 23621 tablet_copy_service.cc:433] P 968c2f6eed3842dfa034660fa0a2117b: ending tablet copy session a3101d9789354f17b9af4da59878eec8-de7005115c824c009aba7de17c8b68ee on tablet de7005115c824c009aba7de17c8b68ee with peer a3101d9789354f17b9af4da59878eec8
I20240502 14:00:42.184460 23599 consensus_queue.cc:237] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.185493 23466 raft_consensus.cc:1261] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240502 14:00:42.186178 25248 consensus_queue.cc:1035] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 14:00:42.187966 25248 raft_consensus.cc:2932] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:42.188494 23468 raft_consensus.cc:2932] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:42.191088 23196 catalog_manager.cc:5484] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index -1 to 92, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
W20240502 14:00:42.213347 23535 consensus_peers.cc:485] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b -> Peer 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367): Couldn't send request to peer 66c6a4ade9e84186beea567c21f80eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78f1ce57b0794e2d9f82d9948ffa9568. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.260731 25344 ts_tablet_manager.cc:902] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:42.261209 25344 tablet_copy_client.cc:308] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:42.262166 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet 1dd0482496c5456c9b491d0d92eecd03 from peer 66c6a4ade9e84186beea567c21f80eb8 ({username='slave'} at 127.16.217.6:40417)
I20240502 14:00:42.262276 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet 1dd0482496c5456c9b491d0d92eecd03 from peer 66c6a4ade9e84186beea567c21f80eb8 at {username='slave'} at 127.16.217.6:40417: session id = 66c6a4ade9e84186beea567c21f80eb8-1dd0482496c5456c9b491d0d92eecd03
I20240502 14:00:42.263193 23359 tablet_copy_source_session.cc:205] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.263980 25344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd0482496c5456c9b491d0d92eecd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.266305 25344 tablet_copy_client.cc:783] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.266443 25344 tablet_copy_client.cc:647] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.268617 25344 tablet_copy_client.cc:523] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.269692 25344 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap starting.
I20240502 14:00:42.274214 25361 ts_tablet_manager.cc:902] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:42.274597 25361 tablet_copy_client.cc:308] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:42.275702 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet dbbf1e7447604baa9701d6330ff31a3f from peer 17a03815fc864e4db7a68c32c6e42616 ({username='slave'} at 127.16.217.5:58087)
I20240502 14:00:42.275789 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet dbbf1e7447604baa9701d6330ff31a3f from peer 17a03815fc864e4db7a68c32c6e42616 at {username='slave'} at 127.16.217.5:58087: session id = 17a03815fc864e4db7a68c32c6e42616-dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:00:42.276453 23359 tablet_copy_source_session.cc:205] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.277169 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf1e7447604baa9701d6330ff31a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.279012 25361 tablet_copy_client.cc:783] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.279143 25361 tablet_copy_client.cc:647] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.282238 25361 tablet_copy_client.cc:523] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.283337 25361 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Bootstrap starting.
I20240502 14:00:42.294046 23599 consensus_queue.cc:237] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.47, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
W20240502 14:00:42.304023 23537 consensus_peers.cc:485] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b -> Peer 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149): Couldn't send request to peer 17a03815fc864e4db7a68c32c6e42616. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87edafbabb594a40b7b20ea97a6e8e51. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.312848 23337 raft_consensus.cc:1261] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 47. Preceding OpId from leader: term: 1 index: 48. (index mismatch)
I20240502 14:00:42.317924 25137 consensus_queue.cc:1035] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 48, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 14:00:42.328351 25147 raft_consensus.cc:2932] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:42.334376 24741 raft_consensus.cc:1203] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.81] Dedup: 1.70->[1.71-1.81]
I20240502 14:00:42.334738 25404 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.334964 25404 tablet_bootstrap.cc:492] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Bootstrap complete.
I20240502 14:00:42.335124 25404 ts_tablet_manager.cc:1367] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.163s user 0.069s sys 0.004s
I20240502 14:00:42.335534 25404 raft_consensus.cc:348] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.335618 25404 raft_consensus.cc:729] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.335752 25404 consensus_queue.cc:260] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.336628 25404 ts_tablet_manager.cc:1398] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:42.337795 23490 tablet_copy_service.cc:342] P 49daee87c26c4b989eaec212705a11f9: Request end of tablet copy session 916fb151c5274f2c8caed395dcdf412b-0b935eeb4b2344d78780f44fff524e9d received from {username='slave'} at 127.16.217.4:41667
I20240502 14:00:42.337881 23490 tablet_copy_service.cc:433] P 49daee87c26c4b989eaec212705a11f9: ending tablet copy session 916fb151c5274f2c8caed395dcdf412b-0b935eeb4b2344d78780f44fff524e9d on tablet 0b935eeb4b2344d78780f44fff524e9d with peer 916fb151c5274f2c8caed395dcdf412b
I20240502 14:00:42.339057 23337 raft_consensus.cc:2932] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Committing config change with OpId 1.48: config changed from index -1 to 48, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:42.341254 23193 catalog_manager.cc:5484] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 48, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:42.376888 24874 raft_consensus.cc:1203] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.88] Dedup: 1.76->[1.77-1.88]
I20240502 14:00:42.386590 25404 ts_tablet_manager.cc:902] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:42.387038 25404 tablet_copy_client.cc:308] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:42.392154 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet d437d8593be743ef8a883ae31d8d5b24 from peer 916fb151c5274f2c8caed395dcdf412b ({username='slave'} at 127.16.217.4:39965)
I20240502 14:00:42.392252 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet d437d8593be743ef8a883ae31d8d5b24 from peer 916fb151c5274f2c8caed395dcdf412b at {username='slave'} at 127.16.217.4:39965: session id = 916fb151c5274f2c8caed395dcdf412b-d437d8593be743ef8a883ae31d8d5b24
I20240502 14:00:42.393144 23359 tablet_copy_source_session.cc:205] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.393903 25404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437d8593be743ef8a883ae31d8d5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.396140 25404 tablet_copy_client.cc:783] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.396922 25404 tablet_copy_client.cc:647] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.397176 23466 consensus_queue.cc:237] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:42.398862 23337 raft_consensus.cc:1261] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
W20240502 14:00:42.400683 23405 consensus_peers.cc:485] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 -> Peer a3101d9789354f17b9af4da59878eec8 (127.16.217.7:32823): Couldn't send request to peer a3101d9789354f17b9af4da59878eec8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 613713c348d64007b0cf596bca97713f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.400975 25104 consensus_queue.cc:1035] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 14:00:42.402325 25404 tablet_copy_client.cc:523] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.403564 25404 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:42.408058 24334 raft_consensus.cc:2932] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:42.408893 25372 ts_tablet_manager.cc:902] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Initiating tablet copy from peer 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:42.409325 25372 tablet_copy_client.cc:308] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:36885
I20240502 14:00:42.410743 23190 catalog_manager.cc:5484] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index -1 to 80, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:42.413480 23337 raft_consensus.cc:2932] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, VOTER a3101d9789354f17b9af4da59878eec8 (127.16.217.7) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:42.416632 25344 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=98 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:42.416893 25344 tablet_bootstrap.cc:492] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap complete.
I20240502 14:00:42.417063 25344 ts_tablet_manager.cc:1367] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Time spent bootstrapping tablet: real 0.147s user 0.079s sys 0.007s
I20240502 14:00:42.417585 25344 raft_consensus.cc:348] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.418061 25344 raft_consensus.cc:729] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.418231 25344 consensus_queue.cc:260] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.418466 25344 ts_tablet_manager.cc:1398] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:00:42.418944 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 66c6a4ade9e84186beea567c21f80eb8-1dd0482496c5456c9b491d0d92eecd03 received from {username='slave'} at 127.16.217.6:40417
I20240502 14:00:42.419018 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 66c6a4ade9e84186beea567c21f80eb8-1dd0482496c5456c9b491d0d92eecd03 on tablet 1dd0482496c5456c9b491d0d92eecd03 with peer 66c6a4ade9e84186beea567c21f80eb8
I20240502 14:00:42.419255 23490 tablet_copy_service.cc:140] P 49daee87c26c4b989eaec212705a11f9: Received BeginTabletCopySession request for tablet 9b29f51fa8f14e3bb8b859b1941e6fac from peer a3101d9789354f17b9af4da59878eec8 ({username='slave'} at 127.16.217.7:34311)
I20240502 14:00:42.419338 23490 tablet_copy_service.cc:161] P 49daee87c26c4b989eaec212705a11f9: Beginning new tablet copy session on tablet 9b29f51fa8f14e3bb8b859b1941e6fac from peer a3101d9789354f17b9af4da59878eec8 at {username='slave'} at 127.16.217.7:34311: session id = a3101d9789354f17b9af4da59878eec8-9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:00:42.420107 23490 tablet_copy_source_session.cc:205] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.420822 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b29f51fa8f14e3bb8b859b1941e6fac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.422876 25372 tablet_copy_client.cc:783] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.423038 25372 tablet_copy_client.cc:647] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.442062 25361 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=761 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.442342 25361 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Bootstrap complete.
I20240502 14:00:42.442516 25361 ts_tablet_manager.cc:1367] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Time spent bootstrapping tablet: real 0.159s user 0.071s sys 0.000s
I20240502 14:00:42.443106 25361 raft_consensus.cc:348] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:42.443236 25361 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a03815fc864e4db7a68c32c6e42616, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.443360 25361 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:42.444576 25372 tablet_copy_client.cc:523] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.445870 25372 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Bootstrap starting.
I20240502 14:00:42.459301 25361 ts_tablet_manager.cc:1398] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240502 14:00:42.461279 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 17a03815fc864e4db7a68c32c6e42616-dbbf1e7447604baa9701d6330ff31a3f received from {username='slave'} at 127.16.217.5:58087
I20240502 14:00:42.461355 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 17a03815fc864e4db7a68c32c6e42616-dbbf1e7447604baa9701d6330ff31a3f on tablet dbbf1e7447604baa9701d6330ff31a3f with peer 17a03815fc864e4db7a68c32c6e42616
I20240502 14:00:42.494102 24608 raft_consensus.cc:1203] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.87] Dedup: 1.75->[1.76-1.87]
I20240502 14:00:42.503520 25008 raft_consensus.cc:1203] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.33->[1.34-1.75] Dedup: 1.55->[1.56-1.75]
I20240502 14:00:42.514293 25404 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=73 ignored=0} inserts{seen=704 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:42.514547 25404 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap complete.
I20240502 14:00:42.514765 25404 ts_tablet_manager.cc:1367] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.111s user 0.067s sys 0.002s
I20240502 14:00:42.515230 25404 raft_consensus.cc:348] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.515528 25404 raft_consensus.cc:729] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.515655 25404 consensus_queue.cc:260] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.515940 25404 ts_tablet_manager.cc:1398] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:42.516856 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 916fb151c5274f2c8caed395dcdf412b-d437d8593be743ef8a883ae31d8d5b24 received from {username='slave'} at 127.16.217.4:39965
I20240502 14:00:42.516929 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 916fb151c5274f2c8caed395dcdf412b-d437d8593be743ef8a883ae31d8d5b24 on tablet d437d8593be743ef8a883ae31d8d5b24 with peer 916fb151c5274f2c8caed395dcdf412b
I20240502 14:00:42.580821 23339 consensus_queue.cc:237] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:42.582383 23599 raft_consensus.cc:1261] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
W20240502 14:00:42.583546 23272 consensus_peers.cc:485] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 -> Peer 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841): Couldn't send request to peer 916fb151c5274f2c8caed395dcdf412b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f625e35721741be82000754b5b1cf2c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.583642 25142 consensus_queue.cc:1035] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 14:00:42.585022 25492 raft_consensus.cc:2932] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.585476 23599 raft_consensus.cc:2932] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.588786 23190 catalog_manager.cc:5484] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 84, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.675034 23601 consensus_queue.cc:237] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.676728 23466 raft_consensus.cc:1261] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 14:00:42.677068 25248 consensus_queue.cc:1035] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 14:00:42.679399 25372 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=724 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.679435 25248 raft_consensus.cc:2932] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:42.679690 25372 tablet_bootstrap.cc:492] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Bootstrap complete.
I20240502 14:00:42.679883 25372 ts_tablet_manager.cc:1367] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Time spent bootstrapping tablet: real 0.234s user 0.080s sys 0.005s
I20240502 14:00:42.680316 25372 raft_consensus.cc:348] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:42.680426 25372 raft_consensus.cc:729] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.680580 25372 consensus_queue.cc:260] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
W20240502 14:00:42.682322 23535 consensus_peers.cc:485] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b -> Peer 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367): Couldn't send request to peer 66c6a4ade9e84186beea567c21f80eb8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e15e8b697e8a4bf8a8b2ac5f9e0022a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.682579 25372 ts_tablet_manager.cc:1398] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:42.683570 23196 catalog_manager.cc:5484] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index -1 to 94, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:42.686506 23490 tablet_copy_service.cc:342] P 49daee87c26c4b989eaec212705a11f9: Request end of tablet copy session a3101d9789354f17b9af4da59878eec8-9b29f51fa8f14e3bb8b859b1941e6fac received from {username='slave'} at 127.16.217.7:34311
I20240502 14:00:42.686581 23490 tablet_copy_service.cc:433] P 49daee87c26c4b989eaec212705a11f9: ending tablet copy session a3101d9789354f17b9af4da59878eec8-9b29f51fa8f14e3bb8b859b1941e6fac on tablet 9b29f51fa8f14e3bb8b859b1941e6fac with peer a3101d9789354f17b9af4da59878eec8
I20240502 14:00:42.688091 23466 raft_consensus.cc:2932] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:42.700261 25344 ts_tablet_manager.cc:902] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Initiating tablet copy from peer 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:42.700736 25344 tablet_copy_client.cc:308] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:33339
I20240502 14:00:42.727612 23621 tablet_copy_service.cc:140] P 968c2f6eed3842dfa034660fa0a2117b: Received BeginTabletCopySession request for tablet 78f1ce57b0794e2d9f82d9948ffa9568 from peer 66c6a4ade9e84186beea567c21f80eb8 ({username='slave'} at 127.16.217.6:38033)
I20240502 14:00:42.727718 23621 tablet_copy_service.cc:161] P 968c2f6eed3842dfa034660fa0a2117b: Beginning new tablet copy session on tablet 78f1ce57b0794e2d9f82d9948ffa9568 from peer 66c6a4ade9e84186beea567c21f80eb8 at {username='slave'} at 127.16.217.6:38033: session id = 66c6a4ade9e84186beea567c21f80eb8-78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:00:42.728751 23621 tablet_copy_source_session.cc:205] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.729681 25344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f1ce57b0794e2d9f82d9948ffa9568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.731416 25344 tablet_copy_client.cc:783] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.731534 25344 tablet_copy_client.cc:647] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.733784 25344 tablet_copy_client.cc:523] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.734761 25344 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap starting.
I20240502 14:00:42.760622 25361 ts_tablet_manager.cc:902] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Initiating tablet copy from peer 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:42.761083 25361 tablet_copy_client.cc:308] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:33339
I20240502 14:00:42.768462 23621 tablet_copy_service.cc:140] P 968c2f6eed3842dfa034660fa0a2117b: Received BeginTabletCopySession request for tablet 87edafbabb594a40b7b20ea97a6e8e51 from peer 17a03815fc864e4db7a68c32c6e42616 ({username='slave'} at 127.16.217.5:52989)
I20240502 14:00:42.768553 23621 tablet_copy_service.cc:161] P 968c2f6eed3842dfa034660fa0a2117b: Beginning new tablet copy session on tablet 87edafbabb594a40b7b20ea97a6e8e51 from peer 17a03815fc864e4db7a68c32c6e42616 at {username='slave'} at 127.16.217.5:52989: session id = 17a03815fc864e4db7a68c32c6e42616-87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:00:42.769500 23621 tablet_copy_source_session.cc:205] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:42.770834 25361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87edafbabb594a40b7b20ea97a6e8e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:42.772925 25361 tablet_copy_client.cc:783] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:42.773065 25361 tablet_copy_client.cc:647] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:42.775012 23337 consensus_queue.cc:237] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
W20240502 14:00:42.776224 23272 consensus_peers.cc:485] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 -> Peer 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841): Couldn't send request to peer 916fb151c5274f2c8caed395dcdf412b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b043771f4c844e86a8d7111f19c45389. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:42.776407 25361 tablet_copy_client.cc:523] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:42.777184 24741 raft_consensus.cc:1203] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.56->[1.57-1.88] Dedup: 1.76->[1.77-1.88]
I20240502 14:00:42.777417 25361 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap starting.
I20240502 14:00:42.777411 23599 raft_consensus.cc:1261] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 14:00:42.777987 25142 consensus_queue.cc:1035] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 14:00:42.783317 25492 raft_consensus.cc:2932] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.785930 23599 raft_consensus.cc:2932] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.785954 23196 catalog_manager.cc:5484] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index -1 to 104, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:42.822247 24874 raft_consensus.cc:1203] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.118] Dedup: 1.99->[1.100-1.118]
I20240502 14:00:42.829628 24608 raft_consensus.cc:1203] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.59->[1.60-1.74] Dedup: 1.74->[]
I20240502 14:00:42.882611 25006 raft_consensus.cc:1203] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.111] Dedup: 1.93->[1.94-1.111]
I20240502 14:00:42.926227 25344 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=98 ignored=0} inserts{seen=770 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:42.926887 25344 tablet_bootstrap.cc:492] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap complete.
I20240502 14:00:42.928117 25344 ts_tablet_manager.cc:1367] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Time spent bootstrapping tablet: real 0.193s user 0.076s sys 0.014s
I20240502 14:00:42.928697 25344 raft_consensus.cc:348] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.928890 25344 raft_consensus.cc:729] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Initialized, Role: FOLLOWER
I20240502 14:00:42.929098 25344 consensus_queue.cc:260] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 98, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:42.931161 25344 ts_tablet_manager.cc:1398] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:00:42.933411 23621 tablet_copy_service.cc:342] P 968c2f6eed3842dfa034660fa0a2117b: Request end of tablet copy session 66c6a4ade9e84186beea567c21f80eb8-78f1ce57b0794e2d9f82d9948ffa9568 received from {username='slave'} at 127.16.217.6:38033
I20240502 14:00:42.933504 23621 tablet_copy_service.cc:433] P 968c2f6eed3842dfa034660fa0a2117b: ending tablet copy session 66c6a4ade9e84186beea567c21f80eb8-78f1ce57b0794e2d9f82d9948ffa9568 on tablet 78f1ce57b0794e2d9f82d9948ffa9568 with peer 66c6a4ade9e84186beea567c21f80eb8
I20240502 14:00:43.017499 25372 ts_tablet_manager.cc:902] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Initiating tablet copy from peer 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:43.018007 25372 tablet_copy_client.cc:308] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:36885
I20240502 14:00:43.018683 23490 tablet_copy_service.cc:140] P 49daee87c26c4b989eaec212705a11f9: Received BeginTabletCopySession request for tablet 613713c348d64007b0cf596bca97713f from peer a3101d9789354f17b9af4da59878eec8 ({username='slave'} at 127.16.217.7:34311)
I20240502 14:00:43.018781 23490 tablet_copy_service.cc:161] P 49daee87c26c4b989eaec212705a11f9: Beginning new tablet copy session on tablet 613713c348d64007b0cf596bca97713f from peer a3101d9789354f17b9af4da59878eec8 at {username='slave'} at 127.16.217.7:34311: session id = a3101d9789354f17b9af4da59878eec8-613713c348d64007b0cf596bca97713f
I20240502 14:00:43.019639 23490 tablet_copy_source_session.cc:205] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:43.019905 25404 ts_tablet_manager.cc:902] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:43.020335 25404 tablet_copy_client.cc:308] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:43.020980 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet 6f625e35721741be82000754b5b1cf2c from peer 916fb151c5274f2c8caed395dcdf412b ({username='slave'} at 127.16.217.4:39965)
I20240502 14:00:43.021070 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet 6f625e35721741be82000754b5b1cf2c from peer 916fb151c5274f2c8caed395dcdf412b at {username='slave'} at 127.16.217.4:39965: session id = 916fb151c5274f2c8caed395dcdf412b-6f625e35721741be82000754b5b1cf2c
I20240502 14:00:43.021831 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613713c348d64007b0cf596bca97713f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.021863 23359 tablet_copy_source_session.cc:205] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:43.022904 25361 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=58 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:43.023247 25361 tablet_bootstrap.cc:492] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap complete.
I20240502 14:00:43.023408 25361 ts_tablet_manager.cc:1367] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Time spent bootstrapping tablet: real 0.246s user 0.051s sys 0.009s
I20240502 14:00:43.024688 25361 raft_consensus.cc:348] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:43.024803 25404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f625e35721741be82000754b5b1cf2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.025195 25361 raft_consensus.cc:729] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a03815fc864e4db7a68c32c6e42616, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.025332 25361 consensus_queue.cc:260] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 58, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 48 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:43.026095 25361 ts_tablet_manager.cc:1398] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:00:43.026712 25404 tablet_copy_client.cc:783] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:43.026835 25404 tablet_copy_client.cc:647] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:43.027616 23621 tablet_copy_service.cc:342] P 968c2f6eed3842dfa034660fa0a2117b: Request end of tablet copy session 17a03815fc864e4db7a68c32c6e42616-87edafbabb594a40b7b20ea97a6e8e51 received from {username='slave'} at 127.16.217.5:52989
I20240502 14:00:43.027678 23621 tablet_copy_service.cc:433] P 968c2f6eed3842dfa034660fa0a2117b: ending tablet copy session 17a03815fc864e4db7a68c32c6e42616-87edafbabb594a40b7b20ea97a6e8e51 on tablet 87edafbabb594a40b7b20ea97a6e8e51 with peer 17a03815fc864e4db7a68c32c6e42616
I20240502 14:00:43.030483 25372 tablet_copy_client.cc:783] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:43.030648 25372 tablet_copy_client.cc:647] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:43.045931 25372 tablet_copy_client.cc:523] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:43.047133 25372 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Bootstrap starting.
I20240502 14:00:43.052515 25404 tablet_copy_client.cc:523] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:43.074717 25404 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:43.095104 24874 raft_consensus.cc:1203] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.91->[1.92-1.107] Dedup: 1.98->[1.99-1.107]
I20240502 14:00:43.195101 25344 ts_tablet_manager.cc:902] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Initiating tablet copy from peer 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339)
I20240502 14:00:43.195569 25344 tablet_copy_client.cc:308] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:33339
I20240502 14:00:43.196786 23621 tablet_copy_service.cc:140] P 968c2f6eed3842dfa034660fa0a2117b: Received BeginTabletCopySession request for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 from peer 66c6a4ade9e84186beea567c21f80eb8 ({username='slave'} at 127.16.217.6:38033)
I20240502 14:00:43.196883 23621 tablet_copy_service.cc:161] P 968c2f6eed3842dfa034660fa0a2117b: Beginning new tablet copy session on tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 from peer 66c6a4ade9e84186beea567c21f80eb8 at {username='slave'} at 127.16.217.6:38033: session id = 66c6a4ade9e84186beea567c21f80eb8-e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:00:43.197790 23621 tablet_copy_source_session.cc:205] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:43.198870 25344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.200891 25344 tablet_copy_client.cc:783] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:43.201025 25344 tablet_copy_client.cc:647] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:43.207648 25344 tablet_copy_client.cc:523] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:43.209017 25344 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap starting.
I20240502 14:00:43.238310 24740 raft_consensus.cc:1203] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.47->[1.48-1.76] Dedup: 1.59->[1.60-1.76]
I20240502 14:00:43.246423 25598 ts_tablet_manager.cc:902] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:43.247009 25598 tablet_copy_client.cc:308] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:00:43.247644 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet b043771f4c844e86a8d7111f19c45389 from peer 916fb151c5274f2c8caed395dcdf412b ({username='slave'} at 127.16.217.4:39965)
I20240502 14:00:43.247756 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet b043771f4c844e86a8d7111f19c45389 from peer 916fb151c5274f2c8caed395dcdf412b at {username='slave'} at 127.16.217.4:39965: session id = 916fb151c5274f2c8caed395dcdf412b-b043771f4c844e86a8d7111f19c45389
I20240502 14:00:43.248806 23359 tablet_copy_source_session.cc:205] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:43.250340 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043771f4c844e86a8d7111f19c45389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:43.263598 25404 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Bootstrap replayed 1/1 log segments. Stats: ops{read=96 overwritten=0 applied=95 ignored=0} inserts{seen=726 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:43.264011 25404 tablet_bootstrap.cc:492] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Bootstrap complete.
I20240502 14:00:43.264202 25404 ts_tablet_manager.cc:1367] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.190s user 0.076s sys 0.010s
I20240502 14:00:43.264667 25404 raft_consensus.cc:348] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:43.265413 25404 raft_consensus.cc:729] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.265781 25404 consensus_queue.cc:260] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 1.96, Last appended by leader: 96, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:43.267647 25404 ts_tablet_manager.cc:1398] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:00:43.269677 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 916fb151c5274f2c8caed395dcdf412b-6f625e35721741be82000754b5b1cf2c received from {username='slave'} at 127.16.217.4:39965
I20240502 14:00:43.269760 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 916fb151c5274f2c8caed395dcdf412b-6f625e35721741be82000754b5b1cf2c on tablet 6f625e35721741be82000754b5b1cf2c with peer 916fb151c5274f2c8caed395dcdf412b
I20240502 14:00:43.270458 25598 tablet_copy_client.cc:783] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:43.271339 25598 tablet_copy_client.cc:647] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:43.316017 25598 tablet_copy_client.cc:523] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:43.320351 25598 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap starting.
I20240502 14:00:43.380915 25344 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=780 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:43.381234 25344 tablet_bootstrap.cc:492] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Bootstrap complete.
I20240502 14:00:43.381433 25344 ts_tablet_manager.cc:1367] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Time spent bootstrapping tablet: real 0.172s user 0.088s sys 0.004s
I20240502 14:00:43.381835 25344 raft_consensus.cc:348] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:43.381954 25344 raft_consensus.cc:729] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.382081 25344 consensus_queue.cc:260] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:43.384589 25344 ts_tablet_manager.cc:1398] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:00:43.387493 23621 tablet_copy_service.cc:342] P 968c2f6eed3842dfa034660fa0a2117b: Request end of tablet copy session 66c6a4ade9e84186beea567c21f80eb8-e15e8b697e8a4bf8a8b2ac5f9e0022a9 received from {username='slave'} at 127.16.217.6:38033
I20240502 14:00:43.387571 23621 tablet_copy_service.cc:433] P 968c2f6eed3842dfa034660fa0a2117b: ending tablet copy session 66c6a4ade9e84186beea567c21f80eb8-e15e8b697e8a4bf8a8b2ac5f9e0022a9 on tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 with peer 66c6a4ade9e84186beea567c21f80eb8
I20240502 14:00:43.421389 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:43.435153 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:43.461721 25372 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:43.462579 25372 tablet_bootstrap.cc:492] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Bootstrap complete.
I20240502 14:00:43.462852 25372 ts_tablet_manager.cc:1367] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Time spent bootstrapping tablet: real 0.416s user 0.064s sys 0.020s
I20240502 14:00:43.463584 25372 raft_consensus.cc:348] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:43.463760 25372 raft_consensus.cc:729] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.463930 25372 consensus_queue.cc:260] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:43.464327 25372 ts_tablet_manager.cc:1398] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:43.465042 23490 tablet_copy_service.cc:342] P 49daee87c26c4b989eaec212705a11f9: Request end of tablet copy session a3101d9789354f17b9af4da59878eec8-613713c348d64007b0cf596bca97713f received from {username='slave'} at 127.16.217.7:34311
I20240502 14:00:43.465123 23490 tablet_copy_service.cc:433] P 49daee87c26c4b989eaec212705a11f9: ending tablet copy session a3101d9789354f17b9af4da59878eec8-613713c348d64007b0cf596bca97713f on tablet 613713c348d64007b0cf596bca97713f with peer a3101d9789354f17b9af4da59878eec8
I20240502 14:00:43.513840 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:43.518513 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:43.519482 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:43.535390 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:43.558372 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:43.607023 24608 raft_consensus.cc:1203] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.83->[1.84-1.107] Dedup: 1.96->[1.97-1.107]
I20240502 14:00:43.654798 24873 raft_consensus.cc:1203] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.93->[1.94-1.122] Dedup: 1.108->[1.109-1.122]
W20240502 14:00:43.701599 23272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.217.3:33339, user_credentials={real_user=slave}} blocked reactor thread for 56584.5us
I20240502 14:00:43.702872 25598 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap replayed 1/1 log segments. Stats: ops{read=120 overwritten=0 applied=120 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:00:43.703246 25598 tablet_bootstrap.cc:492] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Bootstrap complete.
I20240502 14:00:43.703452 25598 ts_tablet_manager.cc:1367] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Time spent bootstrapping tablet: real 0.383s user 0.100s sys 0.004s
I20240502 14:00:43.704572 25598 raft_consensus.cc:348] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:43.704680 25598 raft_consensus.cc:729] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916fb151c5274f2c8caed395dcdf412b, State: Initialized, Role: FOLLOWER
I20240502 14:00:43.704825 25598 consensus_queue.cc:260] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 120, Last appended: 1.120, Last appended by leader: 120, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:43.705083 25598 ts_tablet_manager.cc:1398] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:00:43.705590 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session 916fb151c5274f2c8caed395dcdf412b-b043771f4c844e86a8d7111f19c45389 received from {username='slave'} at 127.16.217.4:39965
I20240502 14:00:43.705664 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session 916fb151c5274f2c8caed395dcdf412b-b043771f4c844e86a8d7111f19c45389 on tablet b043771f4c844e86a8d7111f19c45389 with peer 916fb151c5274f2c8caed395dcdf412b
I20240502 14:00:43.999714 25009 raft_consensus.cc:1203] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.79->[1.80-1.120] Dedup: 1.88->[1.89-1.120]
I20240502 14:00:44.061765 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:44.092033 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:44.127735 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:44.132189 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:44.134342 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:44.143497 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:44.179852 24608 raft_consensus.cc:1203] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.103->[1.104-1.141] Dedup: 1.120->[1.121-1.141]
I20240502 14:00:44.189656 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:44.376695 23468 consensus_queue.cc:237] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:44.377728 25006 raft_consensus.cc:1261] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240502 14:00:44.378113 24334 consensus_queue.cc:1035] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 14:00:44.379997 24334 raft_consensus.cc:2932] T 613713c348d64007b0cf596bca97713f P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 80 to 127, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:44.380337 25008 raft_consensus.cc:2932] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 80 to 127, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:44.382751 23190 catalog_manager.cc:5484] T 613713c348d64007b0cf596bca97713f P a3101d9789354f17b9af4da59878eec8 reported cstate change: config changed from index 80 to 127, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:44.430406 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 613713c348d64007b0cf596bca97713f with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 127) from {username='slave'} at 127.0.0.1:36524
I20240502 14:00:44.436125 25738 tablet_replica.cc:330] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:00:44.436703 25738 raft_consensus.cc:2229] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:44.438803 25738 raft_consensus.cc:2258] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:44.441907 25738 ts_tablet_manager.cc:1875] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:44.454980 25738 ts_tablet_manager.cc:1888] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.126
I20240502 14:00:44.455132 25738 log.cc:1196] T 613713c348d64007b0cf596bca97713f P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/613713c348d64007b0cf596bca97713f
I20240502 14:00:44.457036 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet 613713c348d64007b0cf596bca97713f (table rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]) successfully deleted
I20240502 14:00:44.734867 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:44.748103 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:44.759737 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:44.768774 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:44.806510 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:44.816967 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:44.885754 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:45.066833 23601 consensus_queue.cc:237] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 1.164, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:45.067927 24874 raft_consensus.cc:1261] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20240502 14:00:45.068416 25765 consensus_queue.cc:1035] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20240502 14:00:45.070542 25767 raft_consensus.cc:2932] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.165: config changed from index 94 to 165, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:45.072600 24873 raft_consensus.cc:2932] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 94 to 165, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:45.075208 23196 catalog_manager.cc:5484] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: config changed from index 94 to 165, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:45.116547 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 165) from {username='slave'} at 127.0.0.1:50668
I20240502 14:00:45.125397 25820 tablet_replica.cc:330] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:00:45.126405 25820 raft_consensus.cc:2229] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.126641 25820 raft_consensus.cc:2258] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.127867 25820 ts_tablet_manager.cc:1875] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:45.130939 25820 ts_tablet_manager.cc:1888] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.163
I20240502 14:00:45.131065 25820 log.cc:1196] T e15e8b697e8a4bf8a8b2ac5f9e0022a9 P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:00:45.135499 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9 (table rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]) successfully deleted
I20240502 14:00:45.296397 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:45.297274 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:45.337464 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:45.351143 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:45.373934 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:45.403621 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:45.441946 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:45.648466 23600 consensus_queue.cc:237] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 142, Committed index: 142, Last appended: 1.142, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:45.650161 24741 raft_consensus.cc:1261] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 142. Preceding OpId from leader: term: 1 index: 143. (index mismatch)
I20240502 14:00:45.651150 25765 consensus_queue.cc:1035] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 143, Last known committed idx: 142, Time since last communication: 0.000s
I20240502 14:00:45.654850 25137 raft_consensus.cc:2932] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.143: config changed from index 48 to 143, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:45.659873 23190 catalog_manager.cc:5484] T 87edafbabb594a40b7b20ea97a6e8e51 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index 48 to 143, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:45.660153 24739 raft_consensus.cc:2932] T 87edafbabb594a40b7b20ea97a6e8e51 P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Committing config change with OpId 1.143: config changed from index 48 to 143, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 143 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:45.691550 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 87edafbabb594a40b7b20ea97a6e8e51 with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 143) from {username='slave'} at 127.0.0.1:36524
I20240502 14:00:45.698199 25884 tablet_replica.cc:330] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:00:45.698406 25884 raft_consensus.cc:2229] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:45.698537 25884 raft_consensus.cc:2258] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:45.704238 25884 ts_tablet_manager.cc:1875] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:45.715884 25884 ts_tablet_manager.cc:1888] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.142
I20240502 14:00:45.715963 25884 log.cc:1196] T 87edafbabb594a40b7b20ea97a6e8e51 P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:00:45.716864 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet 87edafbabb594a40b7b20ea97a6e8e51 (table rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]) successfully deleted
I20240502 14:00:46.117197 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:46.122129 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:46.122234 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:46.154614 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.163638 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.193202 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:46.219815 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.331104 23600 consensus_queue.cc:237] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 1.183, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:46.332111 24871 raft_consensus.cc:1261] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240502 14:00:46.332921 25137 consensus_queue.cc:1035] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240502 14:00:46.334867 25767 raft_consensus.cc:2932] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.184: config changed from index 92 to 184, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:46.336596 24874 raft_consensus.cc:2932] T 78f1ce57b0794e2d9f82d9948ffa9568 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index 92 to 184, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:46.337515 23190 catalog_manager.cc:5484] T 78f1ce57b0794e2d9f82d9948ffa9568 P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index 92 to 184, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:46.389667 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f1ce57b0794e2d9f82d9948ffa9568 with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 184) from {username='slave'} at 127.0.0.1:50668
I20240502 14:00:46.398686 25966 tablet_replica.cc:330] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:00:46.398905 25966 raft_consensus.cc:2229] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:46.399089 25966 raft_consensus.cc:2258] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.402220 25966 ts_tablet_manager.cc:1875] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:46.406986 25966 ts_tablet_manager.cc:1888] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.183
I20240502 14:00:46.418458 25966 log.cc:1196] T 78f1ce57b0794e2d9f82d9948ffa9568 P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:00:46.420349 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet 78f1ce57b0794e2d9f82d9948ffa9568 (table rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]) successfully deleted
I20240502 14:00:46.534119 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.534374 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:46.538601 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:46.611057 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.611694 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:46.622421 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:46.644707 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:46.785632 23338 consensus_queue.cc:237] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 1.200, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:46.786711 24608 raft_consensus.cc:1261] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 200. Preceding OpId from leader: term: 1 index: 201. (index mismatch)
I20240502 14:00:46.787169 25144 consensus_queue.cc:1035] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20240502 14:00:46.789760 25142 raft_consensus.cc:2932] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.201: config changed from index 84 to 201, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:46.791527 24605 raft_consensus.cc:2932] T 6f625e35721741be82000754b5b1cf2c P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Committing config change with OpId 1.201: config changed from index 84 to 201, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:46.794415 23196 catalog_manager.cc:5484] T 6f625e35721741be82000754b5b1cf2c P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 84 to 201, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:46.805374 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f625e35721741be82000754b5b1cf2c with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 201) from {username='slave'} at 127.0.0.1:52296
I20240502 14:00:46.807513 26018 tablet_replica.cc:330] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:00:46.808207 26018 raft_consensus.cc:2229] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:46.808990 26018 raft_consensus.cc:2258] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:46.811633 26018 ts_tablet_manager.cc:1875] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:46.813944 26018 ts_tablet_manager.cc:1888] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.200
I20240502 14:00:46.814023 26018 log.cc:1196] T 6f625e35721741be82000754b5b1cf2c P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f625e35721741be82000754b5b1cf2c
I20240502 14:00:46.822351 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet 6f625e35721741be82000754b5b1cf2c (table rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]) successfully deleted
I20240502 14:00:47.129319 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.131995 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:47.132913 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.143735 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.168290 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:47.178218 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:47.196254 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:47.475021 23338 consensus_queue.cc:237] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.198, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:47.476226 24608 raft_consensus.cc:1261] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240502 14:00:47.476920 25792 consensus_queue.cc:1035] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 14:00:47.478331 25792 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.199: config changed from index 60 to 199, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:47.480000 24608 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 60 to 199, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:47.480852 23196 catalog_manager.cc:5484] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 60 to 199, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:47.499603 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet d437d8593be743ef8a883ae31d8d5b24 with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 199) from {username='slave'} at 127.0.0.1:52296
I20240502 14:00:47.506137 26092 tablet_replica.cc:330] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:00:47.506318 26092 raft_consensus.cc:2229] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.529598 26092 raft_consensus.cc:2258] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.530383 26092 ts_tablet_manager.cc:1875] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:47.546452 26092 ts_tablet_manager.cc:1888] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.198
I20240502 14:00:47.546597 26092 log.cc:1196] T d437d8593be743ef8a883ae31d8d5b24 P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/d437d8593be743ef8a883ae31d8d5b24
I20240502 14:00:47.547235 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet d437d8593be743ef8a883ae31d8d5b24 (table rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]) successfully deleted
I20240502 14:00:47.723223 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.738121 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.767515 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:47.804852 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:47.806253 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:47.821835 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:47.821964 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:47.957554 23339 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:47.960014 24741 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 299. Preceding OpId from leader: term: 1 index: 301. (index mismatch)
I20240502 14:00:47.961315 25144 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 299, Time since last communication: 0.000s
I20240502 14:00:47.964655 25142 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 50 to 301, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:47.967568 23196 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 50 to 301, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:47.967932 24739 raft_consensus.cc:1203] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.299->[1.300-1.300] Dedup: 1.300->[]
I20240502 14:00:47.970209 24739 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 50 to 301, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:47.970853 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a4503e308724cddb24e1dd16db817ff with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 301) from {username='slave'} at 127.0.0.1:52296
I20240502 14:00:47.971082 26092 tablet_replica.cc:330] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:00:47.971207 26092 raft_consensus.cc:2229] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:47.971315 26092 raft_consensus.cc:2258] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:47.972103 26092 ts_tablet_manager.cc:1875] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:47.974115 26092 ts_tablet_manager.cc:1888] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.300
I20240502 14:00:47.974176 26092 log.cc:1196] T 1a4503e308724cddb24e1dd16db817ff P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/1a4503e308724cddb24e1dd16db817ff
I20240502 14:00:47.974617 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet 1a4503e308724cddb24e1dd16db817ff (table rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]) successfully deleted
I20240502 14:00:48.208840 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.237005 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:48.237746 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:48.238804 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.250412 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:48.259840 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:48.301342 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.408213 23468 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac"
dest_uuid: "49daee87c26c4b989eaec212705a11f9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52054
I20240502 14:00:48.408406 23468 raft_consensus.cc:595] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:48.441757 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.446420 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.453943 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.463740 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.478423 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.490417 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.501744 23428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.515655 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.527899 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.542260 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.558043 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.572681 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.589159 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
I20240502 14:00:48.598439 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:48.599714 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:00:48.603539 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
I20240502 14:00:48.617336 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.617815 24989 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.618219 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:00:48.624915 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
I20240502 14:00:48.625067 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:48.639927 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:00:48.648856 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.668840 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.687924 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.708386 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.736593 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.760679 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.786850 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.812192 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
W20240502 14:00:48.840837 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Service unavailable: leader transfer in progress
I20240502 14:00:48.845577 25232 raft_consensus.cc:979] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: : Instructing follower a3101d9789354f17b9af4da59878eec8 to start an election
I20240502 14:00:48.845721 25145 raft_consensus.cc:1067] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Signalling peer a3101d9789354f17b9af4da59878eec8 to start an election
I20240502 14:00:48.846264 25006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac"
dest_uuid: "a3101d9789354f17b9af4da59878eec8"
from {username='slave'} at 127.16.217.2:33589
I20240502 14:00:48.846452 25006 raft_consensus.cc:482] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:48.846522 25006 raft_consensus.cc:3037] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:48.847637 25006 raft_consensus.cc:504] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:48.848860 25006 leader_election.cc:288] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [CANDIDATE]: Term 2 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885), 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:48.849403 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac" candidate_uuid: "a3101d9789354f17b9af4da59878eec8" candidate_term: 2 candidate_status { last_received { term: 1 index: 291 } } ignore_live_leader: true dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:48.849498 23470 raft_consensus.cc:3032] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:48.849541 23470 raft_consensus.cc:729] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:48.849681 23470 consensus_queue.cc:260] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 291, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:48.849828 23470 raft_consensus.cc:3037] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:48.850728 23470 raft_consensus.cc:2454] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3101d9789354f17b9af4da59878eec8 in term 2.
I20240502 14:00:48.851056 24941 leader_election.cc:302] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, a3101d9789354f17b9af4da59878eec8; no voters:
I20240502 14:00:48.854200 26258 raft_consensus.cc:2781] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:48.854389 26258 raft_consensus.cc:686] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 2 LEADER]: Becoming Leader. State: Replica: a3101d9789354f17b9af4da59878eec8, State: Running, Role: LEADER
I20240502 14:00:48.855732 26258 consensus_queue.cc:237] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 291, Committed index: 291, Last appended: 1.291, Last appended by leader: 291, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
W20240502 14:00:48.866263 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51824: Illegal state: replica 49daee87c26c4b989eaec212705a11f9 is not leader of this config: current role FOLLOWER
I20240502 14:00:48.868897 23190 catalog_manager.cc:5484] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 reported cstate change: term changed from 1 to 2, leader changed from 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) to a3101d9789354f17b9af4da59878eec8 (127.16.217.7). New cstate: current_term: 2 leader_uuid: "a3101d9789354f17b9af4da59878eec8" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:48.878840 23339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b29f51fa8f14e3bb8b859b1941e6fac" candidate_uuid: "a3101d9789354f17b9af4da59878eec8" candidate_term: 2 candidate_status { last_received { term: 1 index: 291 } } ignore_live_leader: true dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:48.878978 23339 raft_consensus.cc:3037] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:48.880028 23339 raft_consensus.cc:2454] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3101d9789354f17b9af4da59878eec8 in term 2.
I20240502 14:00:48.934209 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.934716 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:48.940122 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:48.958348 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:48.958715 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:48.968114 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:00:48.991963 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50744: Illegal state: replica 6395860ed21149cab200c8364faf6fa8 is not leader of this config: current role FOLLOWER
I20240502 14:00:49.001443 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:49.035672 23470 raft_consensus.cc:1261] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer a3101d9789354f17b9af4da59878eec8: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20240502 14:00:49.035811 23339 raft_consensus.cc:1261] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer a3101d9789354f17b9af4da59878eec8: Log matching property violated. Preceding OpId in replica: term: 1 index: 291. Preceding OpId from leader: term: 2 index: 293. (index mismatch)
I20240502 14:00:49.036121 26258 consensus_queue.cc:1035] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240502 14:00:49.036435 26260 consensus_queue.cc:1035] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 292, Last known committed idx: 291, Time since last communication: 0.000s
I20240502 14:00:49.069022 26286 mvcc.cc:205] Tried to move back new op lower bound from 7023241007244890112 to 7023241006559539200. Current Snapshot: MvccSnapshot[applied={T|T < 7023241007244890112 or (T in {7023241007244890112})}]
I20240502 14:00:49.155588 23601 consensus_queue.cc:237] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 1.308, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:49.171200 25006 raft_consensus.cc:1261] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Refusing update from remote peer 968c2f6eed3842dfa034660fa0a2117b: Log matching property violated. Preceding OpId in replica: term: 1 index: 308. Preceding OpId from leader: term: 1 index: 309. (index mismatch)
I20240502 14:00:49.171680 25765 consensus_queue.cc:1035] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s
I20240502 14:00:49.173874 25765 raft_consensus.cc:2932] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b [term 1 LEADER]: Committing config change with OpId 1.309: config changed from index 34 to 309, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:49.179018 23192 catalog_manager.cc:5484] T de7005115c824c009aba7de17c8b68ee P 968c2f6eed3842dfa034660fa0a2117b reported cstate change: config changed from index 34 to 309, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "968c2f6eed3842dfa034660fa0a2117b" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:49.186003 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet de7005115c824c009aba7de17c8b68ee with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 309) from {username='slave'} at 127.0.0.1:50668
I20240502 14:00:49.186959 26311 tablet_replica.cc:330] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:00:49.187139 26311 raft_consensus.cc:2229] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:49.188438 25008 raft_consensus.cc:2932] T de7005115c824c009aba7de17c8b68ee P a3101d9789354f17b9af4da59878eec8 [term 1 FOLLOWER]: Committing config change with OpId 1.309: config changed from index 34 to 309, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:49.187279 26311 pending_rounds.cc:62] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Trying to abort 1 pending ops.
I20240502 14:00:49.189268 26311 pending_rounds.cc:68] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Aborting op as it isn't in flight: id { term: 1 index: 308 } timestamp: 7023241007708897280 op_type: WRITE_OP write_request { tablet_id: "de7005115c824c009aba7de17c8b68ee" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\340\232\372\020\010K~\274\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""6844bd74f1e4804d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023241007689969664 authz_token { token_data: "\010\264\273\316\261\006\"3\n\005slave\022*\n c634cf930fd44a3caf0750536c6155e5\020\001\030\001 \001(\001" signature: "<redacted>""{\021\227\223>6\340\250,;/\275\206{\243\0232`\371\na\2120\345\242\021;<T\r \2021\241\333F\030\254\323\366Ybk\316B_\022y\361+\264i\347\350 \376\351\262$\217\244%\340\311" signing_key_seq_num: 0 } } request_id { client_id: "5f27d7ff233c47dd990a0572f23bb16c" seq_no: 981 first_incomplete_seq_no: 981 attempt_no: 0 }
I20240502 14:00:49.189759 26311 raft_consensus.cc:2258] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:49.190697 26311 ts_tablet_manager.cc:1875] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:49.199105 26311 ts_tablet_manager.cc:1888] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.308
I20240502 14:00:49.199234 26311 log.cc:1196] T de7005115c824c009aba7de17c8b68ee P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/de7005115c824c009aba7de17c8b68ee
I20240502 14:00:49.202023 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet de7005115c824c009aba7de17c8b68ee (table rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]) successfully deleted
I20240502 14:00:49.457819 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:49.473312 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:49.474781 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:49.483121 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:49.483285 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:49.521232 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:49.523447 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:49.675015 23338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03"
dest_uuid: "6395860ed21149cab200c8364faf6fa8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50786
I20240502 14:00:49.675213 23338 raft_consensus.cc:595] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:49.696118 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.706061 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.713567 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.724643 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.749214 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.765216 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.774277 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.786690 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.804461 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.867611 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.883229 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.898790 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.916297 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.935711 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.957286 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:49.981631 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
W20240502 14:00:50.003683 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
I20240502 14:00:50.008534 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:50.020728 23319 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:00:50.021198 24854 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:50.021752 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.023423 25152 raft_consensus.cc:979] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: : Instructing follower 66c6a4ade9e84186beea567c21f80eb8 to start an election
I20240502 14:00:50.023500 25152 raft_consensus.cc:1067] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Signalling peer 66c6a4ade9e84186beea567c21f80eb8 to start an election
W20240502 14:00:50.025056 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Service unavailable: leader transfer in progress
I20240502 14:00:50.025279 24871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03"
dest_uuid: "66c6a4ade9e84186beea567c21f80eb8"
from {username='slave'} at 127.16.217.1:32785
I20240502 14:00:50.025430 24871 raft_consensus.cc:482] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:50.025517 24871 raft_consensus.cc:3037] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.026495 24871 raft_consensus.cc:504] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.029985 24871 leader_election.cc:288] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [CANDIDATE]: Term 2 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121), 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:00:50.031566 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03" candidate_uuid: "66c6a4ade9e84186beea567c21f80eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: true dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:50.031723 23338 raft_consensus.cc:3032] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:50.031788 23338 raft_consensus.cc:729] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:50.031950 23338 consensus_queue.cc:260] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 345, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.032148 23338 raft_consensus.cc:3037] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.033183 23338 raft_consensus.cc:2454] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c6a4ade9e84186beea567c21f80eb8 in term 2.
I20240502 14:00:50.033854 24808 leader_election.cc:302] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 66c6a4ade9e84186beea567c21f80eb8; no voters:
W20240502 14:00:50.045182 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50400: Illegal state: replica 6395860ed21149cab200c8364faf6fa8 is not leader of this config: current role FOLLOWER
I20240502 14:00:50.049526 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.055799 23470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd0482496c5456c9b491d0d92eecd03" candidate_uuid: "66c6a4ade9e84186beea567c21f80eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 345 } } ignore_live_leader: true dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:50.055918 23470 raft_consensus.cc:3037] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.057005 23470 raft_consensus.cc:2454] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c6a4ade9e84186beea567c21f80eb8 in term 2.
I20240502 14:00:50.059768 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:50.065160 26402 raft_consensus.cc:2781] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:50.065374 26402 raft_consensus.cc:686] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Becoming Leader. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Running, Role: LEADER
I20240502 14:00:50.065675 26402 consensus_queue.cc:237] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 345, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.068104 23190 catalog_manager.cc:5484] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: term changed from 1 to 2, leader changed from 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) to 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6). New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:50.080348 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:00:50.081799 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39418: Illegal state: replica 49daee87c26c4b989eaec212705a11f9 is not leader of this config: current role FOLLOWER
I20240502 14:00:50.117983 23470 raft_consensus.cc:1261] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20240502 14:00:50.118392 26403 consensus_queue.cc:1035] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 14:00:50.121788 23338 raft_consensus.cc:1261] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 2 index: 347. (index mismatch)
I20240502 14:00:50.122197 26071 mvcc.cc:205] Tried to move back new op lower bound from 7023241011679764480 to 7023241011472019456. Current Snapshot: MvccSnapshot[applied={T|T < 7023241011679764480}]
I20240502 14:00:50.124013 26402 consensus_queue.cc:1035] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 14:00:50.350431 24854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.362638 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:50.363945 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.375267 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.380380 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:50.380640 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:50.390584 23319 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:50.522632 23338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74"
dest_uuid: "6395860ed21149cab200c8364faf6fa8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52904
I20240502 14:00:50.522786 23338 raft_consensus.cc:595] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:50.523154 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.542356 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.551910 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.562866 23298 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.571342 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.578819 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.593237 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.603745 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.615713 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.628347 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.645447 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.661588 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.679683 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.695016 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.718844 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.735749 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
I20240502 14:00:50.741878 23319 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:00:50.741922 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.742540 24854 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:50.743338 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:50.745816 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:50.747915 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 14:00:50.757510 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.782271 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
I20240502 14:00:50.802006 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:00:50.805735 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.830650 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.856103 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.882009 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
W20240502 14:00:50.908895 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Service unavailable: leader transfer in progress
I20240502 14:00:50.924794 25152 raft_consensus.cc:979] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: : Instructing follower 66c6a4ade9e84186beea567c21f80eb8 to start an election
I20240502 14:00:50.925045 25142 raft_consensus.cc:1067] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Signalling peer 66c6a4ade9e84186beea567c21f80eb8 to start an election
I20240502 14:00:50.925419 24871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74"
dest_uuid: "66c6a4ade9e84186beea567c21f80eb8"
from {username='slave'} at 127.16.217.1:32785
I20240502 14:00:50.925546 24871 raft_consensus.cc:482] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:50.925601 24871 raft_consensus.cc:3037] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.926713 24871 raft_consensus.cc:504] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.927093 24871 leader_election.cc:288] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [CANDIDATE]: Term 2 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885), 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:00:50.928105 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74" candidate_uuid: "66c6a4ade9e84186beea567c21f80eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 302 } } ignore_live_leader: true dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:50.928211 23338 raft_consensus.cc:3032] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:50.928248 23338 raft_consensus.cc:729] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:50.928395 23338 consensus_queue.cc:260] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 302, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.928560 23338 raft_consensus.cc:3037] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.929622 23338 raft_consensus.cc:2454] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c6a4ade9e84186beea567c21f80eb8 in term 2.
I20240502 14:00:50.930848 24808 leader_election.cc:302] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, 66c6a4ade9e84186beea567c21f80eb8; no voters:
I20240502 14:00:50.931224 26402 raft_consensus.cc:2781] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:50.931306 26402 raft_consensus.cc:686] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Becoming Leader. State: Replica: 66c6a4ade9e84186beea567c21f80eb8, State: Running, Role: LEADER
I20240502 14:00:50.931413 26402 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 302, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:50.932668 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7184d52ecc4a4922bbbd4541eab35d74" candidate_uuid: "66c6a4ade9e84186beea567c21f80eb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 302 } } ignore_live_leader: true dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:00:50.932775 23466 raft_consensus.cc:3037] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:50.932807 23190 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: term changed from 1 to 2, leader changed from 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) to 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6). New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:50.934051 23466 raft_consensus.cc:2454] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66c6a4ade9e84186beea567c21f80eb8 in term 2.
W20240502 14:00:50.941447 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50394: Illegal state: replica 6395860ed21149cab200c8364faf6fa8 is not leader of this config: current role FOLLOWER
W20240502 14:00:50.970139 23430 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51844: Illegal state: replica 49daee87c26c4b989eaec212705a11f9 is not leader of this config: current role FOLLOWER
I20240502 14:00:51.032140 23338 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20240502 14:00:51.032558 26431 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 14:00:51.034785 23466 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 2 index: 304. (index mismatch)
I20240502 14:00:51.040711 26402 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 14:00:51.052999 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:51.074652 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:51.075275 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:51.075697 23319 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:51.082899 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:51.126776 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:51.156569 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:51.283289 23466 consensus_queue.cc:237] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 377, Committed index: 377, Last appended: 1.377, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:51.286373 24607 raft_consensus.cc:1261] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 377. Preceding OpId from leader: term: 1 index: 378. (index mismatch)
I20240502 14:00:51.289587 24334 consensus_queue.cc:1035] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 378, Last known committed idx: 377, Time since last communication: 0.000s
I20240502 14:00:51.291559 25232 raft_consensus.cc:2932] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.378: config changed from index 51 to 378, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:51.296128 24607 raft_consensus.cc:2932] T 0b935eeb4b2344d78780f44fff524e9d P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Committing config change with OpId 1.378: config changed from index 51 to 378, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:51.296903 23190 catalog_manager.cc:5484] T 0b935eeb4b2344d78780f44fff524e9d P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index 51 to 378, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 378 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:51.310292 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b935eeb4b2344d78780f44fff524e9d with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 378) from {username='slave'} at 127.0.0.1:52296
I20240502 14:00:51.325798 26557 tablet_replica.cc:330] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:00:51.325943 26557 raft_consensus.cc:2229] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:51.326062 26557 raft_consensus.cc:2258] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:51.326720 26557 ts_tablet_manager.cc:1875] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:51.330036 26557 ts_tablet_manager.cc:1888] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.377
I20240502 14:00:51.331488 26557 log.cc:1196] T 0b935eeb4b2344d78780f44fff524e9d P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b935eeb4b2344d78780f44fff524e9d
I20240502 14:00:51.350601 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet 0b935eeb4b2344d78780f44fff524e9d (table rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]) successfully deleted
I20240502 14:00:51.628912 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:00:51.642411 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:51.644438 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:51.665346 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:51.701534 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:51.709349 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:51.709823 23319 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:00:51.975975 23339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f"
dest_uuid: "6395860ed21149cab200c8364faf6fa8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52944
I20240502 14:00:51.976190 23339 raft_consensus.cc:595] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Received request to transfer leadership
W20240502 14:00:51.989872 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:51.994755 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:51.999439 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.008853 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.014544 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.025090 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.036931 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.048418 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.061532 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.074781 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.090317 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.106962 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.124825 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.139705 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.146437 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:52.147046 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:00:52.161731 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.164130 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:52.168378 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:00:52.185503 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.191401 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:00:52.206979 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.209249 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:52.210754 23319 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:00:52.227609 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.249333 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.272915 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.297420 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.321458 23297 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.346606 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
W20240502 14:00:52.376927 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.405319 25142 raft_consensus.cc:979] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: : Instructing follower 49daee87c26c4b989eaec212705a11f9 to start an election
I20240502 14:00:52.405426 25142 raft_consensus.cc:1067] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Signalling peer 49daee87c26c4b989eaec212705a11f9 to start an election
I20240502 14:00:52.406494 23466 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f"
dest_uuid: "49daee87c26c4b989eaec212705a11f9"
from {username='slave'} at 127.16.217.1:35417
I20240502 14:00:52.406646 23466 raft_consensus.cc:482] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:00:52.406704 23466 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:00:52.407560 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Service unavailable: leader transfer in progress
I20240502 14:00:52.407745 23466 raft_consensus.cc:504] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:52.408730 23466 leader_election.cc:288] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 2 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121), 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149)
I20240502 14:00:52.409529 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 353 } } ignore_live_leader: true dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:00:52.409688 23336 raft_consensus.cc:3032] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:00:52.409781 23336 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:00:52.409967 23336 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 353, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:52.410511 23336 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:52.413193 23336 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 2.
I20240502 14:00:52.413492 23404 leader_election.cc:302] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, 6395860ed21149cab200c8364faf6fa8; no voters:
I20240502 14:00:52.413941 26600 raft_consensus.cc:2781] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:00:52.414031 26600 raft_consensus.cc:686] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Becoming Leader. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:00:52.414126 26600 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:52.415796 23193 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: term changed from 1 to 2, leader changed from 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) to 49daee87c26c4b989eaec212705a11f9 (127.16.217.2). New cstate: current_term: 2 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:52.417044 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "49daee87c26c4b989eaec212705a11f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 353 } } ignore_live_leader: true dest_uuid: "17a03815fc864e4db7a68c32c6e42616"
I20240502 14:00:52.417228 24739 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:00:52.421605 24739 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49daee87c26c4b989eaec212705a11f9 in term 2.
W20240502 14:00:52.438386 23299 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50370: Illegal state: replica 6395860ed21149cab200c8364faf6fa8 is not leader of this config: current role FOLLOWER
I20240502 14:00:52.484652 23339 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20240502 14:00:52.484668 24741 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 2 index: 355. (index mismatch)
I20240502 14:00:52.486593 25897 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 14:00:52.487497 25897 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 14:00:52.491087 26083 mvcc.cc:205] Tried to move back new op lower bound from 7023241021373403136 to 7023241021089103872. Current Snapshot: MvccSnapshot[applied={T|T < 7023241021373403136 or (T in {7023241021373403136})}]
I20240502 14:00:52.793798 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:52.832612 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:52.835326 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:52.852452 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:52.853292 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:52.853755 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:52.974526 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:53.099148 23339 consensus_queue.cc:237] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 1.397, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } }
I20240502 14:00:53.100515 24608 raft_consensus.cc:1261] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240502 14:00:53.101045 25142 consensus_queue.cc:1035] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240502 14:00:53.102883 25142 raft_consensus.cc:2932] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.398: config changed from index 104 to 398, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:53.105413 23196 catalog_manager.cc:5484] T b043771f4c844e86a8d7111f19c45389 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 104 to 398, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:53.107650 24608 raft_consensus.cc:2932] T b043771f4c844e86a8d7111f19c45389 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 104 to 398, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } }
I20240502 14:00:53.126113 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet b043771f4c844e86a8d7111f19c45389 with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 398) from {username='slave'} at 127.0.0.1:52296
I20240502 14:00:53.126590 26740 tablet_replica.cc:330] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:00:53.126766 26740 raft_consensus.cc:2229] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.126932 26740 raft_consensus.cc:2258] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.133793 26740 ts_tablet_manager.cc:1875] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:53.136602 26740 ts_tablet_manager.cc:1888] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.397
I20240502 14:00:53.136752 26740 log.cc:1196] T b043771f4c844e86a8d7111f19c45389 P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/b043771f4c844e86a8d7111f19c45389
I20240502 14:00:53.137670 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet b043771f4c844e86a8d7111f19c45389 (table rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]) successfully deleted
I20240502 14:00:53.532447 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:53.546752 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:53.560514 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:53.571910 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:53.572175 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:53.593662 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:53.633518 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:53.780328 25007 consensus_queue.cc:237] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 478, Committed index: 478, Last appended: 2.478, Last appended by leader: 291, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } }
I20240502 14:00:53.787021 23336 raft_consensus.cc:1261] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer a3101d9789354f17b9af4da59878eec8: Log matching property violated. Preceding OpId in replica: term: 2 index: 478. Preceding OpId from leader: term: 2 index: 479. (index mismatch)
I20240502 14:00:53.787559 26778 consensus_queue.cc:1035] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 479, Last known committed idx: 478, Time since last communication: 0.000s
I20240502 14:00:53.793898 26778 raft_consensus.cc:2932] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 [term 2 LEADER]: Committing config change with OpId 2.479: config changed from index 71 to 479, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:53.796288 23190 catalog_manager.cc:5484] T 9b29f51fa8f14e3bb8b859b1941e6fac P a3101d9789354f17b9af4da59878eec8 reported cstate change: config changed from index 71 to 479, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "a3101d9789354f17b9af4da59878eec8" committed_config { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:53.800473 23336 raft_consensus.cc:2932] T 9b29f51fa8f14e3bb8b859b1941e6fac P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Committing config change with OpId 2.479: config changed from index 71 to 479, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 479 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "a3101d9789354f17b9af4da59878eec8" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 32823 } } }
I20240502 14:00:53.812502 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b29f51fa8f14e3bb8b859b1941e6fac with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 479) from {username='slave'} at 127.0.0.1:50668
I20240502 14:00:53.812832 26846 tablet_replica.cc:330] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:00:53.813028 26846 raft_consensus.cc:2229] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:53.813421 26846 raft_consensus.cc:2258] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:53.814993 26846 ts_tablet_manager.cc:1875] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:53.817565 26846 ts_tablet_manager.cc:1888] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.478
I20240502 14:00:53.817696 26846 log.cc:1196] T 9b29f51fa8f14e3bb8b859b1941e6fac P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:00:53.818230 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet 9b29f51fa8f14e3bb8b859b1941e6fac (table rebalance_test_table_0 [id=423e92db169441fe8f956155be721b67]) successfully deleted
I20240502 14:00:54.042797 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:54.049543 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.054464 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.061968 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:54.081107 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:54.117125 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:54.117936 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.306936 24872 consensus_queue.cc:237] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 536, Committed index: 536, Last appended: 2.536, Last appended by leader: 345, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:54.308420 23470 raft_consensus.cc:1261] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 536. Preceding OpId from leader: term: 2 index: 537. (index mismatch)
I20240502 14:00:54.309198 26431 consensus_queue.cc:1035] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 537, Last known committed idx: 536, Time since last communication: 0.000s
I20240502 14:00:54.310766 26584 raft_consensus.cc:2932] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Committing config change with OpId 2.537: config changed from index 68 to 537, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.314013 23466 raft_consensus.cc:2932] T 1dd0482496c5456c9b491d0d92eecd03 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Committing config change with OpId 2.537: config changed from index 68 to 537, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.314136 23192 catalog_manager.cc:5484] T 1dd0482496c5456c9b491d0d92eecd03 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: config changed from index 68 to 537, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.361444 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd0482496c5456c9b491d0d92eecd03 with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 537) from {username='slave'} at 127.0.0.1:36524
I20240502 14:00:54.384636 26919 tablet_replica.cc:330] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:00:54.384799 26919 raft_consensus.cc:2229] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:54.385138 26919 raft_consensus.cc:2258] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:54.386435 26919 ts_tablet_manager.cc:1875] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:54.399013 26919 ts_tablet_manager.cc:1888] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.536
I20240502 14:00:54.399168 26919 log.cc:1196] T 1dd0482496c5456c9b491d0d92eecd03 P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/1dd0482496c5456c9b491d0d92eecd03
I20240502 14:00:54.401628 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet 1dd0482496c5456c9b491d0d92eecd03 (table rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]) successfully deleted
I20240502 14:00:54.564807 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:54.590351 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:54.616470 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.618942 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.620421 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:54.626081 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:54.640818 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:54.787068 24872 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 2.430, Last appended by leader: 302, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } }
I20240502 14:00:54.788488 23470 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 430. Preceding OpId from leader: term: 2 index: 431. (index mismatch)
I20240502 14:00:54.789031 26887 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s
I20240502 14:00:54.803540 26887 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Committing config change with OpId 2.431: config changed from index 37 to 431, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.806783 23192 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: config changed from index 37 to 431, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.808591 23466 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Committing config change with OpId 2.431: config changed from index 37 to 431, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } }
I20240502 14:00:54.823374 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 7184d52ecc4a4922bbbd4541eab35d74 with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 431) from {username='slave'} at 127.0.0.1:36524
I20240502 14:00:54.823563 26919 tablet_replica.cc:330] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:00:54.823678 26919 raft_consensus.cc:2229] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:54.823771 26919 raft_consensus.cc:2258] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:54.824877 26919 ts_tablet_manager.cc:1875] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:54.827064 26919 ts_tablet_manager.cc:1888] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.430
I20240502 14:00:54.827131 26919 log.cc:1196] T 7184d52ecc4a4922bbbd4541eab35d74 P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:00:54.832240 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet 7184d52ecc4a4922bbbd4541eab35d74 (table rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]) successfully deleted
I20240502 14:00:55.045480 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:55.063843 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:55.064927 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:55.065896 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.069260 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.075788 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.109203 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:55.285961 23470 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 452, Committed index: 452, Last appended: 2.453, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:55.290412 24740 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 453. Preceding OpId from leader: term: 2 index: 454. (index mismatch)
I20240502 14:00:55.291160 26726 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 452, Time since last communication: 0.000s
I20240502 14:00:55.293171 26726 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Committing config change with OpId 2.454: config changed from index 57 to 454, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:55.295301 24740 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Committing config change with OpId 2.454: config changed from index 57 to 454, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:55.297850 23194 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index 57 to 454, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 2 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:55.301692 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbf1e7447604baa9701d6330ff31a3f with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 454) from {username='slave'} at 127.0.0.1:36524
I20240502 14:00:55.301877 26919 tablet_replica.cc:330] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:00:55.302000 26919 raft_consensus.cc:2229] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:55.302078 26919 pending_rounds.cc:62] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Trying to abort 1 pending ops.
I20240502 14:00:55.302148 26919 pending_rounds.cc:68] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Aborting op as it isn't in flight: id { term: 2 index: 453 } timestamp: 7023241032845324288 op_type: WRITE_OP write_request { tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000$\372\"L\261p\265\375\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""a3eb21f94406e9b6" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023241032808923136 authz_token { token_data: "\010\264\273\316\261\006\"3\n\005slave\022*\n e7b7103fbaaf48dc845e91e9b0b9b502\020\001\030\001 \001(\001" signature: "<redacted>""\232?\2349\252\313 \200g8\311z\246|\252(\251\010\035q\001\265h\310\024}uo\352\226\262.\024\351\036\344\020r=\243\223\250\021\205\024(-;C\016\2174\314\263\2067\017y\217\232:\207p\302" signing_key_seq_num: 0 } } request_id { client_id: "8d7abcfea6974f92a96a88649778b0bf" seq_no: 1392 first_incomplete_seq_no: 1392 attempt_no: 0 }
I20240502 14:00:55.302419 26919 raft_consensus.cc:2258] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:55.303730 26919 ts_tablet_manager.cc:1875] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:55.305881 26919 ts_tablet_manager.cc:1888] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.453
I20240502 14:00:55.305953 26919 log.cc:1196] T dbbf1e7447604baa9701d6330ff31a3f P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:00:55.307070 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet dbbf1e7447604baa9701d6330ff31a3f (table rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]) successfully deleted
I20240502 14:00:55.740744 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.742794 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.762883 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:55.763060 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:55.764515 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:55.775099 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:55.809154 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:55.951718 24872 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 469, Committed index: 469, Last appended: 2.469, Last appended by leader: 302, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:55.954098 23470 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 469. Preceding OpId from leader: term: 2 index: 470. (index mismatch)
I20240502 14:00:55.957661 26547 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 470, Last known committed idx: 469, Time since last communication: 0.000s
I20240502 14:00:55.961509 26988 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Committing config change with OpId 2.470: config changed from index 431 to 470, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:55.962221 23470 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Committing config change with OpId 2.470: config changed from index 431 to 470, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New config: { opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:55.964443 23189 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 reported cstate change: config changed from index 431 to 470, VOTER 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
W20240502 14:00:55.986768 24810 consensus_peers.cc:485] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 -> Peer 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149): Couldn't send request to peer 17a03815fc864e4db7a68c32c6e42616. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7184d52ecc4a4922bbbd4541eab35d74. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:00:56.085892 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:56.087065 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.089500 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:56.090116 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:56.104066 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:56.140496 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.154073 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.556742 27190 ts_tablet_manager.cc:902] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Initiating tablet copy from peer 66c6a4ade9e84186beea567c21f80eb8 (127.16.217.6:42367)
I20240502 14:00:56.557343 27190 tablet_copy_client.cc:308] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.6:42367
I20240502 14:00:56.576047 24895 tablet_copy_service.cc:140] P 66c6a4ade9e84186beea567c21f80eb8: Received BeginTabletCopySession request for tablet 7184d52ecc4a4922bbbd4541eab35d74 from peer 17a03815fc864e4db7a68c32c6e42616 ({username='slave'} at 127.16.217.5:52605)
I20240502 14:00:56.576181 24895 tablet_copy_service.cc:161] P 66c6a4ade9e84186beea567c21f80eb8: Beginning new tablet copy session on tablet 7184d52ecc4a4922bbbd4541eab35d74 from peer 17a03815fc864e4db7a68c32c6e42616 at {username='slave'} at 127.16.217.5:52605: session id = 17a03815fc864e4db7a68c32c6e42616-7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:00:56.577742 24895 tablet_copy_source_session.cc:205] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:00:56.578892 27190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7184d52ecc4a4922bbbd4541eab35d74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:00:56.581195 27190 tablet_copy_client.cc:783] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 0 data blocks...
I20240502 14:00:56.581346 27190 tablet_copy_client.cc:647] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Starting download of 1 WAL segments...
I20240502 14:00:56.593281 27190 tablet_copy_client.cc:523] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:00:56.594574 27190 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap starting.
I20240502 14:00:56.641433 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:56.656499 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.664348 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:56.679612 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:56.686177 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.688872 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:56.696424 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:57.116261 27190 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap replayed 1/1 log segments. Stats: ops{read=492 overwritten=0 applied=491 ignored=0} inserts{seen=1173 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:00:57.116710 27190 tablet_bootstrap.cc:492] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Bootstrap complete.
I20240502 14:00:57.117055 27190 ts_tablet_manager.cc:1367] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Time spent bootstrapping tablet: real 0.523s user 0.311s sys 0.044s
I20240502 14:00:57.117578 27190 raft_consensus.cc:348] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:57.117987 27190 raft_consensus.cc:729] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a03815fc864e4db7a68c32c6e42616, State: Initialized, Role: FOLLOWER
I20240502 14:00:57.118136 27190 consensus_queue.cc:260] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 491, Last appended: 2.492, Last appended by leader: 492, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 470 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:57.118458 27190 ts_tablet_manager.cc:1398] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:00:57.119355 24895 tablet_copy_service.cc:342] P 66c6a4ade9e84186beea567c21f80eb8: Request end of tablet copy session 17a03815fc864e4db7a68c32c6e42616-7184d52ecc4a4922bbbd4541eab35d74 received from {username='slave'} at 127.16.217.5:52605
I20240502 14:00:57.119446 24895 tablet_copy_service.cc:433] P 66c6a4ade9e84186beea567c21f80eb8: ending tablet copy session 17a03815fc864e4db7a68c32c6e42616-7184d52ecc4a4922bbbd4541eab35d74 on tablet 7184d52ecc4a4922bbbd4541eab35d74 with peer 17a03815fc864e4db7a68c32c6e42616
I20240502 14:00:57.234745 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:57.237396 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.259730 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:57.264551 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.279302 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.287909 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:57.292553 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:57.456076 24739 raft_consensus.cc:1203] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.469->[2.470-2.509] Dedup: 2.492->[2.493-2.509]
I20240502 14:00:57.828488 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:57.832232 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:57.844609 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:57.849165 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.856339 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.873530 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:57.900390 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:58.085868 24871 consensus_queue.cc:237] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 535, Committed index: 535, Last appended: 2.536, Last appended by leader: 302, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } }
I20240502 14:00:58.087348 24738 raft_consensus.cc:1261] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer 66c6a4ade9e84186beea567c21f80eb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 536. Preceding OpId from leader: term: 2 index: 537. (index mismatch)
I20240502 14:00:58.087932 26988 consensus_queue.cc:1035] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 537, Last known committed idx: 535, Time since last communication: 0.000s
I20240502 14:00:58.089638 26988 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 66c6a4ade9e84186beea567c21f80eb8 [term 2 LEADER]: Committing config change with OpId 2.537: config changed from index 470 to 537, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:58.094089 24738 raft_consensus.cc:2932] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Committing config change with OpId 2.537: config changed from index 470 to 537, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:58.107080 23193 catalog_manager.cc:5484] T 7184d52ecc4a4922bbbd4541eab35d74 P 17a03815fc864e4db7a68c32c6e42616 reported cstate change: config changed from index 470 to 537, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "66c6a4ade9e84186beea567c21f80eb8" committed_config { opid_index: 537 OBSOLETE_local: false peers { permanent_uuid: "66c6a4ade9e84186beea567c21f80eb8" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 42367 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } }
I20240502 14:00:58.134169 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet 7184d52ecc4a4922bbbd4541eab35d74 with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 537) from {username='slave'} at 127.0.0.1:50668
I20240502 14:00:58.134517 27339 tablet_replica.cc:330] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:00:58.134676 27339 raft_consensus.cc:2229] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:00:58.134794 27339 pending_rounds.cc:62] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Trying to abort 1 pending ops.
I20240502 14:00:58.134846 27339 pending_rounds.cc:68] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Aborting op as it isn't in flight: id { term: 2 index: 536 } timestamp: 7023241044294520832 op_type: WRITE_OP write_request { tablet_id: "7184d52ecc4a4922bbbd4541eab35d74" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000 \253\220\013\265\311;8\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""f857be3a91141f1f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023241044079341568 authz_token { token_data: "\010\264\273\316\261\006\"3\n\005slave\022*\n 77f446494d9942fda9cbf833f760be3f\020\001\030\001 \001(\001" signature: "<redacted>""\014\270D\177\267\200\033\233\273_4\313\231i\344\211_\254\330\"=\017#\010F=\272g\325#w\1776\343\375\272 \311]i\276\013W\356\004@\333\250\270\'\220\203\2102\034\335\003\202\257\314\315\245\027\231" signing_key_seq_num: 0 } } request_id { client_id: "08d13a3b6fc046fdb1f826211851b967" seq_no: 1547 first_incomplete_seq_no: 1547 attempt_no: 0 }
I20240502 14:00:58.135134 27339 raft_consensus.cc:2258] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:00:58.136022 27339 ts_tablet_manager.cc:1875] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:00:58.138957 27339 ts_tablet_manager.cc:1888] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.536
I20240502 14:00:58.139031 27339 log.cc:1196] T 7184d52ecc4a4922bbbd4541eab35d74 P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:00:58.140250 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet 7184d52ecc4a4922bbbd4541eab35d74 (table rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]) successfully deleted
I20240502 14:00:58.489503 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:58.526050 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.537992 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:58.538347 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.549942 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:58.550146 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:58.556531 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.826835 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.857687 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:58.877650 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:58.878319 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.884656 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:58.891397 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:58.907181 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:59.646056 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:59.674914 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:59.675563 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:00:59.749239 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:00:59.757012 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:00:59.782115 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:00:59.818317 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:00.095150 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:00.106936 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.109054 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.123060 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.139307 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:00.156157 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:00.167282 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:00.346205 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:00.347810 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.349655 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:00.350298 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:00.350494 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.352521 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:00.367638 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
c7eeb47ae899472f8cb9c37a97c5c29d | 127.16.217.62:32783 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.217.20:43383 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
17a03815fc864e4db7a68c32c6e42616 | 127.16.217.5:43149 | HEALTHY | <none> | 0 | 0
49daee87c26c4b989eaec212705a11f9 | 127.16.217.2:36885 | HEALTHY | <none> | 4 | 0
6395860ed21149cab200c8364faf6fa8 | 127.16.217.1:35121 | HEALTHY | <none> | 4 | 0
66c6a4ade9e84186beea567c21f80eb8 | 127.16.217.6:42367 | HEALTHY | <none> | 2 | 0
916fb151c5274f2c8caed395dcdf412b | 127.16.217.4:41841 | HEALTHY | <none> | 1 | 0
968c2f6eed3842dfa034660fa0a2117b | 127.16.217.3:33339 | HEALTHY | <none> | 4 | 0
a3101d9789354f17b9af4da59878eec8 | 127.16.217.7:32823 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.16.217.1 | experimental | 127.16.217.1:35121
local_ip_for_outbound_sockets | 127.16.217.2 | experimental | 127.16.217.2:36885
local_ip_for_outbound_sockets | 127.16.217.3 | experimental | 127.16.217.3:33339
local_ip_for_outbound_sockets | 127.16.217.4 | experimental | 127.16.217.4:41841
local_ip_for_outbound_sockets | 127.16.217.5 | experimental | 127.16.217.5:43149
local_ip_for_outbound_sockets | 127.16.217.6 | experimental | 127.16.217.6:42367
local_ip_for_outbound_sockets | 127.16.217.7 | experimental | 127.16.217.7:32823
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.217.1:35121
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.217.2:36885
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.217.3:33339
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.217.4:41841
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.217.5:43149
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.217.6:42367
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.217.7:32823
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.217.20:43383 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:01:00.393910 17252 log_verifier.cc:126] Checking tablet 0b935eeb4b2344d78780f44fff524e9d
I20240502 14:01:00.501510 17252 log_verifier.cc:177] Verified matching terms for 739 ops in tablet 0b935eeb4b2344d78780f44fff524e9d
I20240502 14:01:00.501763 17252 log_verifier.cc:126] Checking tablet 1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:00.618445 17252 log_verifier.cc:177] Verified matching terms for 810 ops in tablet 1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:00.618690 17252 log_verifier.cc:126] Checking tablet 1dd0482496c5456c9b491d0d92eecd03
I20240502 14:01:00.730780 17252 log_verifier.cc:177] Verified matching terms for 793 ops in tablet 1dd0482496c5456c9b491d0d92eecd03
I20240502 14:01:00.731024 17252 log_verifier.cc:126] Checking tablet 5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:00.818904 17252 log_verifier.cc:177] Verified matching terms for 579 ops in tablet 5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:00.819096 17252 log_verifier.cc:126] Checking tablet 613713c348d64007b0cf596bca97713f
I20240502 14:01:00.915769 17252 log_verifier.cc:177] Verified matching terms for 657 ops in tablet 613713c348d64007b0cf596bca97713f
I20240502 14:01:00.916044 17252 log_verifier.cc:126] Checking tablet 6f625e35721741be82000754b5b1cf2c
I20240502 14:01:01.006824 17252 log_verifier.cc:177] Verified matching terms for 621 ops in tablet 6f625e35721741be82000754b5b1cf2c
I20240502 14:01:01.006979 17252 log_verifier.cc:126] Checking tablet 7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:01:01.095748 17252 log_verifier.cc:177] Verified matching terms for 614 ops in tablet 7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:01:01.095944 17252 log_verifier.cc:126] Checking tablet 78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:01:01.193346 17252 log_verifier.cc:177] Verified matching terms for 615 ops in tablet 78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:01:01.193540 17252 log_verifier.cc:126] Checking tablet 87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:01:01.294188 17252 log_verifier.cc:177] Verified matching terms for 703 ops in tablet 87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:01:01.294374 17252 log_verifier.cc:126] Checking tablet 9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:01:01.400177 17252 log_verifier.cc:177] Verified matching terms for 728 ops in tablet 9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:01:01.400405 17252 log_verifier.cc:126] Checking tablet 9eb5ab6a504d4fceab781c331a40627d
I20240502 14:01:01.413571 17252 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9eb5ab6a504d4fceab781c331a40627d
I20240502 14:01:01.413642 17252 log_verifier.cc:126] Checking tablet b043771f4c844e86a8d7111f19c45389
I20240502 14:01:01.512643 17252 log_verifier.cc:177] Verified matching terms for 681 ops in tablet b043771f4c844e86a8d7111f19c45389
I20240502 14:01:01.512836 17252 log_verifier.cc:126] Checking tablet d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:01.602998 17252 log_verifier.cc:177] Verified matching terms for 606 ops in tablet d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:01.603173 17252 log_verifier.cc:126] Checking tablet dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:01.712879 17252 log_verifier.cc:177] Verified matching terms for 652 ops in tablet dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:01.713078 17252 log_verifier.cc:126] Checking tablet de7005115c824c009aba7de17c8b68ee
I20240502 14:01:01.825908 17252 log_verifier.cc:177] Verified matching terms for 805 ops in tablet de7005115c824c009aba7de17c8b68ee
I20240502 14:01:01.826222 17252 log_verifier.cc:126] Checking tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:01:01.923187 17252 log_verifier.cc:177] Verified matching terms for 658 ops in tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:01:01.925345 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.8:0
--local_ip_for_outbound_sockets=127.16.217.8
--webserver_interface=127.16.217.8
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--builtin_ntp_servers=127.16.217.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:01:02.026129 27570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:02.026324 27570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:02.026386 27570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:02.029136 27570 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:02.029192 27570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:02.029269 27570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.8
I20240502 14:01:02.032371 27570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.217.8
--webserver_port=0
--tserver_master_addrs=127.16.217.62:32783
--never_fsync=true
--heap_profile_path=/tmp/kudu.27570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.8
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:02.033128 27570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:02.034153 27570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:02.041028 27576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:02.041566 27575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:02.042079 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:02.042824 27570 server_base.cc:1034] running on GCE node
I20240502 14:01:02.043119 27570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:02.043618 27570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:02.044759 27570 hybrid_clock.cc:648] HybridClock initialized: now 1714658462044750 us; error 23 us; skew 500 ppm
I20240502 14:01:02.050740 27570 webserver.cc:460] Webserver started at http://127.16.217.8:42191/ using document root <none> and password file <none>
I20240502 14:01:02.051211 27570 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:02.051259 27570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:02.051421 27570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:02.052883 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f596ef9abe2d445f86dfe24ea58036fa"
format_stamp: "Formatted at 2024-05-02 14:01:02 on dist-test-slave-7msf"
I20240502 14:01:02.053289 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f596ef9abe2d445f86dfe24ea58036fa"
format_stamp: "Formatted at 2024-05-02 14:01:02 on dist-test-slave-7msf"
I20240502 14:01:02.056087 27570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:01:02.057925 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:02.058905 27570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:02.058993 27570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f596ef9abe2d445f86dfe24ea58036fa"
format_stamp: "Formatted at 2024-05-02 14:01:02 on dist-test-slave-7msf"
I20240502 14:01:02.059062 27570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:02.068488 27570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:02.068982 27570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:02.069119 27570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:02.069568 27570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:02.070331 27570 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:02.070374 27570 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:02.070421 27570 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:02.070451 27570 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:02.077684 27570 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.8:39833
I20240502 14:01:02.077710 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.8:39833 every 8 connection(s)
I20240502 14:01:02.078428 27570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:01:02.080986 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 27570
I20240502 14:01:02.081101 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:01:02.086238 27699 heartbeater.cc:347] Connected to a master server at 127.16.217.62:32783
I20240502 14:01:02.086473 27699 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:02.086957 27699 heartbeater.cc:511] Master 127.16.217.62:32783 requested a full tablet report, sending...
I20240502 14:01:02.087929 23193 ts_manager.cc:194] Registered new tserver with Master: f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8:39833)
I20240502 14:01:02.088481 23193 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.8:44327
I20240502 14:01:02.258932 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.259099 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.268883 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.272629 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.272833 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.276371 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.294662 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.308873 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.377838 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.382355 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.383246 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.385143 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.390967 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.395527 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.398510 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.400053 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.444710 23466 consensus_queue.cc:237] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 579, Committed index: 579, Last appended: 1.579, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:02.446048 23600 raft_consensus.cc:1261] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 579. Preceding OpId from leader: term: 1 index: 580. (index mismatch)
I20240502 14:01:02.446434 27805 consensus_queue.cc:1035] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 580, Last known committed idx: 579, Time since last communication: 0.000s
I20240502 14:01:02.448292 27805 raft_consensus.cc:2932] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.580: config changed from index -1 to 580, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.449894 23600 raft_consensus.cc:2932] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Committing config change with OpId 1.580: config changed from index -1 to 580, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.451191 23192 catalog_manager.cc:5484] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index -1 to 580, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
W20240502 14:01:02.452229 23406 consensus_peers.cc:485] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 -> Peer f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8:39833): Couldn't send request to peer f596ef9abe2d445f86dfe24ea58036fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a11bbc1c922463e9c73acc2966972ac. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.468176 23336 consensus_queue.cc:237] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 606, Committed index: 606, Last appended: 1.606, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:02.469605 24608 raft_consensus.cc:1261] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 606. Preceding OpId from leader: term: 1 index: 607. (index mismatch)
I20240502 14:01:02.469995 27828 consensus_queue.cc:1035] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 607, Last known committed idx: 606, Time since last communication: 0.000s
I20240502 14:01:02.471630 25144 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.607: config changed from index 199 to 607, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
W20240502 14:01:02.472429 23275 consensus_peers.cc:485] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 -> Peer f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8:39833): Couldn't send request to peer f596ef9abe2d445f86dfe24ea58036fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d437d8593be743ef8a883ae31d8d5b24. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.472681 24608 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Committing config change with OpId 1.607: config changed from index 199 to 607, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.474143 23193 catalog_manager.cc:5484] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 199 to 607, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.494591 23336 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 810, Committed index: 810, Last appended: 1.810, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:02.495887 24739 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 810. Preceding OpId from leader: term: 1 index: 811. (index mismatch)
W20240502 14:01:02.496158 23275 consensus_peers.cc:485] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 -> Peer f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8:39833): Couldn't send request to peer f596ef9abe2d445f86dfe24ea58036fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a4503e308724cddb24e1dd16db817ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.496374 25144 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 811, Last known committed idx: 810, Time since last communication: 0.000s
I20240502 14:01:02.497952 27828 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.811: config changed from index 301 to 811, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.498343 24739 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Committing config change with OpId 1.811: config changed from index 301 to 811, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.500770 23193 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 reported cstate change: config changed from index 301 to 811, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.517982 23466 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 652, Committed index: 652, Last appended: 2.652, Last appended by leader: 353, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:02.519263 24739 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 652. Preceding OpId from leader: term: 2 index: 653. (index mismatch)
W20240502 14:01:02.519282 23406 consensus_peers.cc:485] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 -> Peer f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8:39833): Couldn't send request to peer f596ef9abe2d445f86dfe24ea58036fa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbbf1e7447604baa9701d6330ff31a3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:02.519634 27341 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 653, Last known committed idx: 652, Time since last communication: 0.000s
I20240502 14:01:02.521142 27805 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Committing config change with OpId 2.653: config changed from index 454 to 653, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.521517 24739 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Committing config change with OpId 2.653: config changed from index 454 to 653, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New config: { opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.523795 23192 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 reported cstate change: config changed from index 454 to 653, VOTER f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8) added. New cstate: current_term: 2 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:02.576941 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.580165 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.585640 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.592538 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.593125 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.593787 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.600499 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.604630 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.676985 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.676985 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.682410 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.687109 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.688952 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.690255 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.693521 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.701578 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.782773 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.783216 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.784142 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.788370 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.789268 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.789541 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.800223 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.817889 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.825892 27980 ts_tablet_manager.cc:902] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Initiating tablet copy from peer 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:01:02.826726 27980 tablet_copy_client.cc:308] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:36885
I20240502 14:01:02.832640 23490 tablet_copy_service.cc:140] P 49daee87c26c4b989eaec212705a11f9: Received BeginTabletCopySession request for tablet 5a11bbc1c922463e9c73acc2966972ac from peer f596ef9abe2d445f86dfe24ea58036fa ({username='slave'} at 127.16.217.8:46851)
I20240502 14:01:02.832774 23490 tablet_copy_service.cc:161] P 49daee87c26c4b989eaec212705a11f9: Beginning new tablet copy session on tablet 5a11bbc1c922463e9c73acc2966972ac from peer f596ef9abe2d445f86dfe24ea58036fa at {username='slave'} at 127.16.217.8:46851: session id = f596ef9abe2d445f86dfe24ea58036fa-5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:02.833552 23490 tablet_copy_source_session.cc:205] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:02.836647 27980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a11bbc1c922463e9c73acc2966972ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:02.842816 27980 tablet_copy_client.cc:783] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:02.843063 27980 tablet_copy_client.cc:647] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:02.847196 27980 tablet_copy_client.cc:523] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:02.849159 27980 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap starting.
I20240502 14:01:02.897627 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.899873 27980 log.cc:824] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:02.903183 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.910329 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.911211 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:02.915830 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:02.917209 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:02.924346 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:02.930322 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.008482 28059 ts_tablet_manager.cc:902] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:01:03.009150 28059 tablet_copy_client.cc:308] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:01:03.016768 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.017395 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.023316 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.025017 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet 1a4503e308724cddb24e1dd16db817ff from peer f596ef9abe2d445f86dfe24ea58036fa ({username='slave'} at 127.16.217.8:50809)
I20240502 14:01:03.025120 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet 1a4503e308724cddb24e1dd16db817ff from peer f596ef9abe2d445f86dfe24ea58036fa at {username='slave'} at 127.16.217.8:50809: session id = f596ef9abe2d445f86dfe24ea58036fa-1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:03.025966 23359 tablet_copy_source_session.cc:205] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:03.026858 28059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a4503e308724cddb24e1dd16db817ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:03.029039 28059 tablet_copy_client.cc:783] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:03.029187 28059 tablet_copy_client.cc:647] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:03.032208 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:03.032893 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.034652 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.043299 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.044371 28059 tablet_copy_client.cc:523] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:03.045876 28059 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap starting.
I20240502 14:01:03.063417 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.075183 28065 ts_tablet_manager.cc:902] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Initiating tablet copy from peer 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121)
I20240502 14:01:03.075675 28065 tablet_copy_client.cc:308] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:35121
I20240502 14:01:03.076241 23359 tablet_copy_service.cc:140] P 6395860ed21149cab200c8364faf6fa8: Received BeginTabletCopySession request for tablet d437d8593be743ef8a883ae31d8d5b24 from peer f596ef9abe2d445f86dfe24ea58036fa ({username='slave'} at 127.16.217.8:50809)
I20240502 14:01:03.076364 23359 tablet_copy_service.cc:161] P 6395860ed21149cab200c8364faf6fa8: Beginning new tablet copy session on tablet d437d8593be743ef8a883ae31d8d5b24 from peer f596ef9abe2d445f86dfe24ea58036fa at {username='slave'} at 127.16.217.8:50809: session id = f596ef9abe2d445f86dfe24ea58036fa-d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:03.077234 23359 tablet_copy_source_session.cc:205] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:03.077991 28065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d437d8593be743ef8a883ae31d8d5b24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:03.080265 28065 tablet_copy_client.cc:783] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:03.080565 28065 tablet_copy_client.cc:647] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:03.089092 28065 tablet_copy_client.cc:523] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:03.090274 28065 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap starting.
I20240502 14:01:03.090577 27699 heartbeater.cc:503] Master 127.16.217.62:32783 was elected leader, sending a full tablet report...
I20240502 14:01:03.099306 28067 ts_tablet_manager.cc:902] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Initiating tablet copy from peer 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885)
I20240502 14:01:03.099848 28067 tablet_copy_client.cc:308] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:36885
I20240502 14:01:03.100354 23490 tablet_copy_service.cc:140] P 49daee87c26c4b989eaec212705a11f9: Received BeginTabletCopySession request for tablet dbbf1e7447604baa9701d6330ff31a3f from peer f596ef9abe2d445f86dfe24ea58036fa ({username='slave'} at 127.16.217.8:46851)
I20240502 14:01:03.100476 23490 tablet_copy_service.cc:161] P 49daee87c26c4b989eaec212705a11f9: Beginning new tablet copy session on tablet dbbf1e7447604baa9701d6330ff31a3f from peer f596ef9abe2d445f86dfe24ea58036fa at {username='slave'} at 127.16.217.8:46851: session id = f596ef9abe2d445f86dfe24ea58036fa-dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:03.101374 23490 tablet_copy_source_session.cc:205] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:03.106998 28067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbbf1e7447604baa9701d6330ff31a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:03.109923 28067 tablet_copy_client.cc:783] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:03.110071 28067 tablet_copy_client.cc:647] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:03.112871 28067 tablet_copy_client.cc:523] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:03.114040 28067 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap starting.
I20240502 14:01:03.297494 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.310052 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.335582 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.336997 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:03.366883 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.377173 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.393538 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.410953 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.414301 27980 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=580 overwritten=0 applied=580 ignored=0} inserts{seen=1252 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:03.414747 27980 tablet_bootstrap.cc:492] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap complete.
I20240502 14:01:03.415150 27980 ts_tablet_manager.cc:1367] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Time spent bootstrapping tablet: real 0.566s user 0.367s sys 0.021s
I20240502 14:01:03.418270 27980 raft_consensus.cc:348] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.418548 27980 raft_consensus.cc:729] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Initialized, Role: FOLLOWER
I20240502 14:01:03.419078 27980 consensus_queue.cc:260] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 580, Last appended: 1.580, Last appended by leader: 580, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 580 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "968c2f6eed3842dfa034660fa0a2117b" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 33339 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.425990 27980 ts_tablet_manager.cc:1398] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240502 14:01:03.427488 23490 tablet_copy_service.cc:342] P 49daee87c26c4b989eaec212705a11f9: Request end of tablet copy session f596ef9abe2d445f86dfe24ea58036fa-5a11bbc1c922463e9c73acc2966972ac received from {username='slave'} at 127.16.217.8:46851
I20240502 14:01:03.427603 23490 tablet_copy_service.cc:433] P 49daee87c26c4b989eaec212705a11f9: ending tablet copy session f596ef9abe2d445f86dfe24ea58036fa-5a11bbc1c922463e9c73acc2966972ac on tablet 5a11bbc1c922463e9c73acc2966972ac with peer f596ef9abe2d445f86dfe24ea58036fa
I20240502 14:01:03.569540 28065 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=607 overwritten=0 applied=607 ignored=0} inserts{seen=1236 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:03.569855 28065 tablet_bootstrap.cc:492] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap complete.
I20240502 14:01:03.570114 28065 ts_tablet_manager.cc:1367] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Time spent bootstrapping tablet: real 0.480s user 0.352s sys 0.034s
I20240502 14:01:03.570668 28065 raft_consensus.cc:348] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.570775 28065 raft_consensus.cc:729] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Initialized, Role: FOLLOWER
I20240502 14:01:03.570905 28065 consensus_queue.cc:260] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 607, Last appended: 1.607, Last appended by leader: 607, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 607 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "916fb151c5274f2c8caed395dcdf412b" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 41841 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.571302 28065 ts_tablet_manager.cc:1398] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:01:03.572185 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session f596ef9abe2d445f86dfe24ea58036fa-d437d8593be743ef8a883ae31d8d5b24 received from {username='slave'} at 127.16.217.8:50809
I20240502 14:01:03.572291 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session f596ef9abe2d445f86dfe24ea58036fa-d437d8593be743ef8a883ae31d8d5b24 on tablet d437d8593be743ef8a883ae31d8d5b24 with peer f596ef9abe2d445f86dfe24ea58036fa
I20240502 14:01:03.642607 28067 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=653 overwritten=0 applied=653 ignored=0} inserts{seen=1335 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:03.642937 28067 tablet_bootstrap.cc:492] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap complete.
I20240502 14:01:03.643178 28067 ts_tablet_manager.cc:1367] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Time spent bootstrapping tablet: real 0.529s user 0.402s sys 0.028s
I20240502 14:01:03.643671 28067 raft_consensus.cc:348] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.643793 28067 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Initialized, Role: FOLLOWER
I20240502 14:01:03.643929 28067 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 653, Last appended: 2.653, Last appended by leader: 653, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.644244 28067 ts_tablet_manager.cc:1398] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:03.644842 23490 tablet_copy_service.cc:342] P 49daee87c26c4b989eaec212705a11f9: Request end of tablet copy session f596ef9abe2d445f86dfe24ea58036fa-dbbf1e7447604baa9701d6330ff31a3f received from {username='slave'} at 127.16.217.8:46851
I20240502 14:01:03.644932 23490 tablet_copy_service.cc:433] P 49daee87c26c4b989eaec212705a11f9: ending tablet copy session f596ef9abe2d445f86dfe24ea58036fa-dbbf1e7447604baa9701d6330ff31a3f on tablet dbbf1e7447604baa9701d6330ff31a3f with peer f596ef9abe2d445f86dfe24ea58036fa
I20240502 14:01:03.671499 27652 raft_consensus.cc:1203] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.652->[2.653-2.653] Dedup: 2.653->[]
I20240502 14:01:03.709568 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:03.714309 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.716406 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.716670 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.735038 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.741864 28059 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=811 overwritten=0 applied=811 ignored=0} inserts{seen=1440 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:03.742188 28059 tablet_bootstrap.cc:492] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Bootstrap complete.
I20240502 14:01:03.742455 28059 ts_tablet_manager.cc:1367] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Time spent bootstrapping tablet: real 0.697s user 0.494s sys 0.034s
I20240502 14:01:03.742585 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.742928 28059 raft_consensus.cc:348] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.743069 28059 raft_consensus.cc:729] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Initialized, Role: FOLLOWER
I20240502 14:01:03.743201 28059 consensus_queue.cc:260] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 811, Last appended: 1.811, Last appended by leader: 811, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:03.743603 28059 ts_tablet_manager.cc:1398] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:03.744308 23359 tablet_copy_service.cc:342] P 6395860ed21149cab200c8364faf6fa8: Request end of tablet copy session f596ef9abe2d445f86dfe24ea58036fa-1a4503e308724cddb24e1dd16db817ff received from {username='slave'} at 127.16.217.8:50809
I20240502 14:01:03.744395 23359 tablet_copy_service.cc:433] P 6395860ed21149cab200c8364faf6fa8: ending tablet copy session f596ef9abe2d445f86dfe24ea58036fa-1a4503e308724cddb24e1dd16db817ff on tablet 1a4503e308724cddb24e1dd16db817ff with peer f596ef9abe2d445f86dfe24ea58036fa
I20240502 14:01:03.750386 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.752358 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.816133 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.818840 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:03.819250 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.828507 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.836452 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.837281 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.843180 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.850524 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.904497 27652 raft_consensus.cc:1203] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.579->[1.580-1.580] Dedup: 1.580->[]
I20240502 14:01:03.923130 27652 raft_consensus.cc:1203] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.810->[1.811-1.811] Dedup: 1.811->[]
I20240502 14:01:03.923825 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.926946 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.931860 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.934933 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:03.937250 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:03.940691 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.955824 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:03.966192 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:03.985395 23466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f"
dest_uuid: "49daee87c26c4b989eaec212705a11f9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46748
I20240502 14:01:03.985545 23466 raft_consensus.cc:595] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Received request to transfer leadership
I20240502 14:01:03.998983 27652 raft_consensus.cc:1203] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.606->[1.607-1.607] Dedup: 1.607->[]
I20240502 14:01:04.033747 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.034269 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.039772 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.051460 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.052167 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.052431 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.053007 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.057257 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.143882 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.147727 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.148043 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.148469 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.152479 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.153546 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.158934 27632 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.162406 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.212926 23466 consensus_queue.cc:237] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 580, Committed index: 580, Last appended: 1.580, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 581 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.214074 27652 raft_consensus.cc:1261] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Refusing update from remote peer 49daee87c26c4b989eaec212705a11f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 580. Preceding OpId from leader: term: 1 index: 581. (index mismatch)
I20240502 14:01:04.214457 28066 consensus_queue.cc:1035] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 581, Last known committed idx: 580, Time since last communication: 0.000s
I20240502 14:01:04.216840 28324 raft_consensus.cc:2932] T 5a11bbc1c922463e9c73acc2966972ac P 49daee87c26c4b989eaec212705a11f9 [term 1 LEADER]: Committing config change with OpId 1.581: config changed from index 580 to 581, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 581 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.217191 27652 raft_consensus.cc:2932] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Committing config change with OpId 1.581: config changed from index 580 to 581, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New config: { opid_index: 581 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.219651 23193 catalog_manager.cc:5484] T 5a11bbc1c922463e9c73acc2966972ac P f596ef9abe2d445f86dfe24ea58036fa reported cstate change: config changed from index 580 to 581, VOTER 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "49daee87c26c4b989eaec212705a11f9" committed_config { opid_index: 581 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.222419 23581 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a11bbc1c922463e9c73acc2966972ac with delete_type TABLET_DATA_TOMBSTONED (TS 968c2f6eed3842dfa034660fa0a2117b not found in new config with opid_index 581) from {username='slave'} at 127.0.0.1:52296
I20240502 14:01:04.222748 28334 tablet_replica.cc:330] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: stopping tablet replica
I20240502 14:01:04.222879 28334 raft_consensus.cc:2229] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.223019 28334 raft_consensus.cc:2258] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:04.224463 28334 ts_tablet_manager.cc:1875] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.226887 28334 ts_tablet_manager.cc:1888] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.580
I20240502 14:01:04.226972 28334 log.cc:1196] T 5a11bbc1c922463e9c73acc2966972ac P 968c2f6eed3842dfa034660fa0a2117b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:04.227638 23181 catalog_manager.cc:4830] TS 968c2f6eed3842dfa034660fa0a2117b (127.16.217.3:33339): tablet 5a11bbc1c922463e9c73acc2966972ac (table rebalance_test_table_3 [id=77f446494d9942fda9cbf833f760be3f]) successfully deleted
I20240502 14:01:04.272423 28324 raft_consensus.cc:979] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: : Instructing follower f596ef9abe2d445f86dfe24ea58036fa to start an election
I20240502 14:01:04.272555 28066 raft_consensus.cc:1067] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Signalling peer f596ef9abe2d445f86dfe24ea58036fa to start an election
I20240502 14:01:04.273679 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.273797 27652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f"
dest_uuid: "f596ef9abe2d445f86dfe24ea58036fa"
from {username='slave'} at 127.16.217.2:33395
I20240502 14:01:04.273990 27652 raft_consensus.cc:482] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:04.274106 27652 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:04.274804 23450 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.275292 27652 raft_consensus.cc:504] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.276739 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "f596ef9abe2d445f86dfe24ea58036fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 653 } } ignore_live_leader: true dest_uuid: "49daee87c26c4b989eaec212705a11f9"
I20240502 14:01:04.276841 23466 raft_consensus.cc:3032] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:01:04.276875 23466 raft_consensus.cc:729] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 49daee87c26c4b989eaec212705a11f9, State: Running, Role: LEADER
I20240502 14:01:04.276994 23466 consensus_queue.cc:260] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 653, Committed index: 653, Last appended: 2.653, Last appended by leader: 653, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.277128 23466 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:04.278045 23466 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f596ef9abe2d445f86dfe24ea58036fa in term 3.
I20240502 14:01:04.278381 27587 leader_election.cc:302] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49daee87c26c4b989eaec212705a11f9, f596ef9abe2d445f86dfe24ea58036fa; no voters:
I20240502 14:01:04.278628 28117 raft_consensus.cc:2781] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:04.278781 28117 raft_consensus.cc:686] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 3 LEADER]: Becoming Leader. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Running, Role: LEADER
I20240502 14:01:04.279073 28117 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 653, Committed index: 653, Last appended: 2.653, Last appended by leader: 653, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.281287 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.281478 23194 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa reported cstate change: term changed from 2 to 3, leader changed from 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) to f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8). New cstate: current_term: 3 leader_uuid: "f596ef9abe2d445f86dfe24ea58036fa" committed_config { opid_index: 653 OBSOLETE_local: false peers { permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.279834 27652 leader_election.cc:288] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [CANDIDATE]: Term 3 election: Requested vote from peers 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885), 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149)
I20240502 14:01:04.284837 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.285357 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.288489 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.302156 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbbf1e7447604baa9701d6330ff31a3f" candidate_uuid: "f596ef9abe2d445f86dfe24ea58036fa" candidate_term: 3 candidate_status { last_received { term: 2 index: 653 } } ignore_live_leader: true dest_uuid: "17a03815fc864e4db7a68c32c6e42616"
I20240502 14:01:04.302285 24739 raft_consensus.cc:3037] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:04.303234 24739 raft_consensus.cc:2454] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f596ef9abe2d445f86dfe24ea58036fa in term 3.
I20240502 14:01:04.304946 27632 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.312937 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.352216 23336 consensus_queue.cc:237] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 607, Committed index: 607, Last appended: 1.607, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.353274 27652 raft_consensus.cc:1261] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Refusing update from remote peer 6395860ed21149cab200c8364faf6fa8: Log matching property violated. Preceding OpId in replica: term: 1 index: 607. Preceding OpId from leader: term: 1 index: 608. (index mismatch)
I20240502 14:01:04.353637 25144 consensus_queue.cc:1035] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 608, Last known committed idx: 607, Time since last communication: 0.000s
I20240502 14:01:04.355240 28389 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Committing config change with OpId 1.608: config changed from index 607 to 608, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) evicted. New config: { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.355644 27652 raft_consensus.cc:2932] T d437d8593be743ef8a883ae31d8d5b24 P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Committing config change with OpId 1.608: config changed from index 607 to 608, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) evicted. New config: { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.357687 23194 catalog_manager.cc:5484] T d437d8593be743ef8a883ae31d8d5b24 P 6395860ed21149cab200c8364faf6fa8 reported cstate change: config changed from index 607 to 608, VOTER 916fb151c5274f2c8caed395dcdf412b (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "6395860ed21149cab200c8364faf6fa8" committed_config { opid_index: 608 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.362838 24588 tablet_service.cc:1514] Processing DeleteTablet for tablet d437d8593be743ef8a883ae31d8d5b24 with delete_type TABLET_DATA_TOMBSTONED (TS 916fb151c5274f2c8caed395dcdf412b not found in new config with opid_index 608) from {username='slave'} at 127.0.0.1:33438
I20240502 14:01:04.363287 28391 tablet_replica.cc:330] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: stopping tablet replica
I20240502 14:01:04.363543 28391 raft_consensus.cc:2229] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.363725 28391 raft_consensus.cc:2258] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:04.365345 28391 ts_tablet_manager.cc:1875] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.367388 28391 ts_tablet_manager.cc:1888] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.607
I20240502 14:01:04.367457 28391 log.cc:1196] T d437d8593be743ef8a883ae31d8d5b24 P 916fb151c5274f2c8caed395dcdf412b: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:04.368053 23181 catalog_manager.cc:4830] TS 916fb151c5274f2c8caed395dcdf412b (127.16.217.4:41841): tablet d437d8593be743ef8a883ae31d8d5b24 (table rebalance_test_table_2 [id=a096e94ba545420d9e6632b84be791a0]) successfully deleted
I20240502 14:01:04.613142 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.613538 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:04.619340 27632 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.623284 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.627820 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.629884 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.640555 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.656782 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.677624 23336 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff"
dest_uuid: "6395860ed21149cab200c8364faf6fa8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37782
I20240502 14:01:04.677757 23336 raft_consensus.cc:595] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:04.711900 23466 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 3 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 653. Preceding OpId from leader: term: 3 index: 654. (index mismatch)
I20240502 14:01:04.712945 28370 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "49daee87c26c4b989eaec212705a11f9" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 654, Last known committed idx: 653, Time since last communication: 0.000s
I20240502 14:01:04.718075 24739 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 3 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 653. Preceding OpId from leader: term: 3 index: 654. (index mismatch)
I20240502 14:01:04.718526 28117 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 654, Last known committed idx: 653, Time since last communication: 0.000s
I20240502 14:01:04.727536 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.728508 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:04.738766 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.740216 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.741190 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.742254 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.751933 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.752444 27632 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.833680 23319 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.834545 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.835580 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.841135 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.844486 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:04.862612 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:04.865566 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.870571 27632 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.904773 27652 consensus_queue.cc:237] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 654, Committed index: 654, Last appended: 3.654, Last appended by leader: 653, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 655 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.905767 24739 raft_consensus.cc:1261] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 3 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 3 index: 654. Preceding OpId from leader: term: 3 index: 655. (index mismatch)
I20240502 14:01:04.906088 28117 consensus_queue.cc:1035] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 655, Last known committed idx: 654, Time since last communication: 0.000s
I20240502 14:01:04.907493 28389 raft_consensus.cc:979] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: : Instructing follower f596ef9abe2d445f86dfe24ea58036fa to start an election
I20240502 14:01:04.907591 28389 raft_consensus.cc:1067] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Signalling peer f596ef9abe2d445f86dfe24ea58036fa to start an election
I20240502 14:01:04.907699 28370 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa [term 3 LEADER]: Committing config change with OpId 3.655: config changed from index 653 to 655, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 655 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.908089 24739 raft_consensus.cc:2932] T dbbf1e7447604baa9701d6330ff31a3f P 17a03815fc864e4db7a68c32c6e42616 [term 3 FOLLOWER]: Committing config change with OpId 3.655: config changed from index 653 to 655, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New config: { opid_index: 655 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.909662 27652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff"
dest_uuid: "f596ef9abe2d445f86dfe24ea58036fa"
from {username='slave'} at 127.16.217.1:40325
I20240502 14:01:04.909770 27652 raft_consensus.cc:482] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:04.909832 27652 raft_consensus.cc:3037] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.910035 23189 catalog_manager.cc:5484] T dbbf1e7447604baa9701d6330ff31a3f P f596ef9abe2d445f86dfe24ea58036fa reported cstate change: config changed from index 653 to 655, VOTER 49daee87c26c4b989eaec212705a11f9 (127.16.217.2) evicted. New cstate: current_term: 3 leader_uuid: "f596ef9abe2d445f86dfe24ea58036fa" committed_config { opid_index: 655 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.910667 27652 raft_consensus.cc:504] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.911037 27652 leader_election.cc:288] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [CANDIDATE]: Term 2 election: Requested vote from peers 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121), 17a03815fc864e4db7a68c32c6e42616 (127.16.217.5:43149)
I20240502 14:01:04.911660 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff" candidate_uuid: "f596ef9abe2d445f86dfe24ea58036fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 811 } } ignore_live_leader: true dest_uuid: "6395860ed21149cab200c8364faf6fa8"
I20240502 14:01:04.911780 23336 raft_consensus.cc:3032] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:04.911821 23336 raft_consensus.cc:729] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6395860ed21149cab200c8364faf6fa8, State: Running, Role: LEADER
I20240502 14:01:04.911964 23336 consensus_queue.cc:260] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 811, Committed index: 811, Last appended: 1.811, Last appended by leader: 811, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.912130 23336 raft_consensus.cc:3037] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.912302 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a4503e308724cddb24e1dd16db817ff" candidate_uuid: "f596ef9abe2d445f86dfe24ea58036fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 811 } } ignore_live_leader: true dest_uuid: "17a03815fc864e4db7a68c32c6e42616"
I20240502 14:01:04.912387 24739 raft_consensus.cc:3037] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:04.913081 23336 raft_consensus.cc:2454] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f596ef9abe2d445f86dfe24ea58036fa in term 2.
I20240502 14:01:04.913185 24739 raft_consensus.cc:2454] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f596ef9abe2d445f86dfe24ea58036fa in term 2.
I20240502 14:01:04.913414 27586 leader_election.cc:302] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6395860ed21149cab200c8364faf6fa8, f596ef9abe2d445f86dfe24ea58036fa; no voters:
I20240502 14:01:04.913653 28370 raft_consensus.cc:2781] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:04.913771 28370 raft_consensus.cc:686] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 2 LEADER]: Becoming Leader. State: Replica: f596ef9abe2d445f86dfe24ea58036fa, State: Running, Role: LEADER
I20240502 14:01:04.913887 28370 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 811, Committed index: 811, Last appended: 1.811, Last appended by leader: 811, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:04.914438 23450 tablet_service.cc:1514] Processing DeleteTablet for tablet dbbf1e7447604baa9701d6330ff31a3f with delete_type TABLET_DATA_TOMBSTONED (TS 49daee87c26c4b989eaec212705a11f9 not found in new config with opid_index 655) from {username='slave'} at 127.0.0.1:50668
I20240502 14:01:04.914984 28538 tablet_replica.cc:330] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: stopping tablet replica
I20240502 14:01:04.915136 28538 raft_consensus.cc:2229] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:04.915241 28538 raft_consensus.cc:2258] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:04.915342 23196 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa reported cstate change: term changed from 1 to 2, leader changed from 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) to f596ef9abe2d445f86dfe24ea58036fa (127.16.217.8). New cstate: current_term: 2 leader_uuid: "f596ef9abe2d445f86dfe24ea58036fa" committed_config { opid_index: 811 OBSOLETE_local: false peers { permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 } } peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:04.917577 28538 ts_tablet_manager.cc:1875] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:04.919888 28538 ts_tablet_manager.cc:1888] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.654
I20240502 14:01:04.919965 28538 log.cc:1196] T dbbf1e7447604baa9701d6330ff31a3f P 49daee87c26c4b989eaec212705a11f9: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:04.920598 23183 catalog_manager.cc:4830] TS 49daee87c26c4b989eaec212705a11f9 (127.16.217.2:36885): tablet dbbf1e7447604baa9701d6330ff31a3f (table rebalance_test_table_1 [id=e7b7103fbaaf48dc845e91e9b0b9b502]) successfully deleted
I20240502 14:01:04.976538 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:04.978296 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:04.983067 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.983788 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:04.985854 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:04.986020 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:04.988020 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.013478 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:05.089848 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:05.090961 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.094144 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.097784 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.110185 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.116037 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:05.124430 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.124872 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.402978 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.404042 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.409029 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.410833 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.411191 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.415284 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:05.415635 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:05.433259 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.455333 23336 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 811. Preceding OpId from leader: term: 2 index: 812. (index mismatch)
I20240502 14:01:05.455854 28667 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6395860ed21149cab200c8364faf6fa8" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 812, Last known committed idx: 811, Time since last communication: 0.000s
I20240502 14:01:05.458531 24739 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 811. Preceding OpId from leader: term: 2 index: 812. (index mismatch)
I20240502 14:01:05.459055 28667 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 812, Last known committed idx: 811, Time since last communication: 0.000s
I20240502 14:01:05.707415 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:05.713558 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.714104 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.715399 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.721369 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:05.721626 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:05.743356 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:05.745822 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:05.778342 27652 consensus_queue.cc:237] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 812, Committed index: 812, Last appended: 2.812, Last appended by leader: 811, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 813 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } }
I20240502 14:01:05.779512 24739 raft_consensus.cc:1261] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Refusing update from remote peer f596ef9abe2d445f86dfe24ea58036fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 812. Preceding OpId from leader: term: 2 index: 813. (index mismatch)
I20240502 14:01:05.779922 28667 consensus_queue.cc:1035] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 813, Last known committed idx: 812, Time since last communication: 0.000s
I20240502 14:01:05.781312 28669 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P f596ef9abe2d445f86dfe24ea58036fa [term 2 LEADER]: Committing config change with OpId 2.813: config changed from index 811 to 813, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 813 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:05.781683 24739 raft_consensus.cc:2932] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 [term 2 FOLLOWER]: Committing config change with OpId 2.813: config changed from index 811 to 813, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New config: { opid_index: 813 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:05.785121 23194 catalog_manager.cc:5484] T 1a4503e308724cddb24e1dd16db817ff P 17a03815fc864e4db7a68c32c6e42616 reported cstate change: config changed from index 811 to 813, VOTER 6395860ed21149cab200c8364faf6fa8 (127.16.217.1) evicted. New cstate: current_term: 2 leader_uuid: "f596ef9abe2d445f86dfe24ea58036fa" committed_config { opid_index: 813 OBSOLETE_local: false peers { permanent_uuid: "17a03815fc864e4db7a68c32c6e42616" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 43149 } } peers { permanent_uuid: "f596ef9abe2d445f86dfe24ea58036fa" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 39833 } } }
I20240502 14:01:05.787531 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a4503e308724cddb24e1dd16db817ff with delete_type TABLET_DATA_TOMBSTONED (TS 6395860ed21149cab200c8364faf6fa8 not found in new config with opid_index 813) from {username='slave'} at 127.0.0.1:36524
I20240502 14:01:05.787889 28734 tablet_replica.cc:330] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: stopping tablet replica
I20240502 14:01:05.788025 28734 raft_consensus.cc:2229] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:05.788127 28734 raft_consensus.cc:2258] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:05.789408 28734 ts_tablet_manager.cc:1875] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:05.791209 28734 ts_tablet_manager.cc:1888] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.812
I20240502 14:01:05.791270 28734 log.cc:1196] T 1a4503e308724cddb24e1dd16db817ff P 6395860ed21149cab200c8364faf6fa8: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:05.791824 23182 catalog_manager.cc:4830] TS 6395860ed21149cab200c8364faf6fa8 (127.16.217.1:35121): tablet 1a4503e308724cddb24e1dd16db817ff (table rebalance_test_table_4 [id=c634cf930fd44a3caf0750536c6155e5]) successfully deleted
I20240502 14:01:06.043524 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:06.043810 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:06.048045 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.048476 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:06.053053 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.063089 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:06.063109 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.085655 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.162673 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:06.164691 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:06.164954 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.169878 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.170326 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:06.170475 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.198776 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:06.203305 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.282629 24854 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.286231 24588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:06.295244 24721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:06.302611 23581 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:06.303362 23450 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.314623 27632 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:06.316460 23319 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:06.326767 24989 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
c7eeb47ae899472f8cb9c37a97c5c29d | 127.16.217.62:32783 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.217.20:43383 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
17a03815fc864e4db7a68c32c6e42616 | 127.16.217.5:43149 | HEALTHY | <none> | 0 | 0
49daee87c26c4b989eaec212705a11f9 | 127.16.217.2:36885 | HEALTHY | <none> | 3 | 0
6395860ed21149cab200c8364faf6fa8 | 127.16.217.1:35121 | HEALTHY | <none> | 3 | 0
66c6a4ade9e84186beea567c21f80eb8 | 127.16.217.6:42367 | HEALTHY | <none> | 2 | 0
916fb151c5274f2c8caed395dcdf412b | 127.16.217.4:41841 | HEALTHY | <none> | 1 | 0
968c2f6eed3842dfa034660fa0a2117b | 127.16.217.3:33339 | HEALTHY | <none> | 4 | 0
a3101d9789354f17b9af4da59878eec8 | 127.16.217.7:32823 | HEALTHY | <none> | 1 | 0
f596ef9abe2d445f86dfe24ea58036fa | 127.16.217.8:39833 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.16.217.1 | experimental | 127.16.217.1:35121
local_ip_for_outbound_sockets | 127.16.217.2 | experimental | 127.16.217.2:36885
local_ip_for_outbound_sockets | 127.16.217.3 | experimental | 127.16.217.3:33339
local_ip_for_outbound_sockets | 127.16.217.4 | experimental | 127.16.217.4:41841
local_ip_for_outbound_sockets | 127.16.217.5 | experimental | 127.16.217.5:43149
local_ip_for_outbound_sockets | 127.16.217.6 | experimental | 127.16.217.6:42367
local_ip_for_outbound_sockets | 127.16.217.7 | experimental | 127.16.217.7:32823
local_ip_for_outbound_sockets | 127.16.217.8 | experimental | 127.16.217.8:39833
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.217.1:35121
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.217.2:36885
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.217.3:33339
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.217.4:41841
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.217.5:43149
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.217.6:42367
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.217.7:32823
server_dump_info_path | /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.16.217.8:39833
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.217.20:43383 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:01:06.349651 17252 log_verifier.cc:126] Checking tablet 0b935eeb4b2344d78780f44fff524e9d
I20240502 14:01:06.484773 17252 log_verifier.cc:177] Verified matching terms for 739 ops in tablet 0b935eeb4b2344d78780f44fff524e9d
I20240502 14:01:06.485237 17252 log_verifier.cc:126] Checking tablet 1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:06.692843 17252 log_verifier.cc:177] Verified matching terms for 813 ops in tablet 1a4503e308724cddb24e1dd16db817ff
I20240502 14:01:06.693094 17252 log_verifier.cc:126] Checking tablet 1dd0482496c5456c9b491d0d92eecd03
I20240502 14:01:06.809525 17252 log_verifier.cc:177] Verified matching terms for 793 ops in tablet 1dd0482496c5456c9b491d0d92eecd03
I20240502 14:01:06.809760 17252 log_verifier.cc:126] Checking tablet 5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:06.914168 17252 log_verifier.cc:177] Verified matching terms for 581 ops in tablet 5a11bbc1c922463e9c73acc2966972ac
I20240502 14:01:06.914340 17252 log_verifier.cc:126] Checking tablet 613713c348d64007b0cf596bca97713f
I20240502 14:01:07.011895 17252 log_verifier.cc:177] Verified matching terms for 657 ops in tablet 613713c348d64007b0cf596bca97713f
I20240502 14:01:07.012082 17252 log_verifier.cc:126] Checking tablet 6f625e35721741be82000754b5b1cf2c
I20240502 14:01:07.103220 17252 log_verifier.cc:177] Verified matching terms for 621 ops in tablet 6f625e35721741be82000754b5b1cf2c
I20240502 14:01:07.103382 17252 log_verifier.cc:126] Checking tablet 7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:01:07.195523 17252 log_verifier.cc:177] Verified matching terms for 614 ops in tablet 7184d52ecc4a4922bbbd4541eab35d74
I20240502 14:01:07.195724 17252 log_verifier.cc:126] Checking tablet 78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:01:07.285492 17252 log_verifier.cc:177] Verified matching terms for 615 ops in tablet 78f1ce57b0794e2d9f82d9948ffa9568
I20240502 14:01:07.285682 17252 log_verifier.cc:126] Checking tablet 87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:01:07.382833 17252 log_verifier.cc:177] Verified matching terms for 703 ops in tablet 87edafbabb594a40b7b20ea97a6e8e51
I20240502 14:01:07.383066 17252 log_verifier.cc:126] Checking tablet 9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:01:07.484405 17252 log_verifier.cc:177] Verified matching terms for 728 ops in tablet 9b29f51fa8f14e3bb8b859b1941e6fac
I20240502 14:01:07.484634 17252 log_verifier.cc:126] Checking tablet 9eb5ab6a504d4fceab781c331a40627d
I20240502 14:01:07.493468 17252 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9eb5ab6a504d4fceab781c331a40627d
I20240502 14:01:07.493525 17252 log_verifier.cc:126] Checking tablet b043771f4c844e86a8d7111f19c45389
I20240502 14:01:07.593303 17252 log_verifier.cc:177] Verified matching terms for 681 ops in tablet b043771f4c844e86a8d7111f19c45389
I20240502 14:01:07.593540 17252 log_verifier.cc:126] Checking tablet d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:07.698161 17252 log_verifier.cc:177] Verified matching terms for 608 ops in tablet d437d8593be743ef8a883ae31d8d5b24
I20240502 14:01:07.698367 17252 log_verifier.cc:126] Checking tablet dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:07.801193 17252 log_verifier.cc:177] Verified matching terms for 655 ops in tablet dbbf1e7447604baa9701d6330ff31a3f
I20240502 14:01:07.801424 17252 log_verifier.cc:126] Checking tablet de7005115c824c009aba7de17c8b68ee
I20240502 14:01:07.921599 17252 log_verifier.cc:177] Verified matching terms for 805 ops in tablet de7005115c824c009aba7de17c8b68ee
I20240502 14:01:07.921830 17252 log_verifier.cc:126] Checking tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:01:08.021095 17252 log_verifier.cc:177] Verified matching terms for 658 ops in tablet e15e8b697e8a4bf8a8b2ac5f9e0022a9
I20240502 14:01:08.022660 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23257
I20240502 14:01:08.030995 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23388
I20240502 14:01:08.038105 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23519
W20240502 14:01:08.044143 24807 connection.cc:535] server connection from 127.16.217.3:58333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:01:08.044692 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 24525
I20240502 14:01:08.050616 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 24658
I20240502 14:01:08.057055 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 24791
I20240502 14:01:08.062703 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 24924
I20240502 14:01:08.068712 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 27570
I20240502 14:01:08.074347 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 23166
2024-05-02T14:01:08Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (36999 ms)
[----------] 1 test from RebalanceParamTest (36999 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 14:01:08.131505 17252 test_util.cc:268] Using random seed: 149385898
I20240502 14:01:08.132341 17252 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:01:08.132381 17252 ts_itest-base.cc:113] --------------
I20240502 14:01:08.132401 17252 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:01:08.132416 17252 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:01:08.132433 17252 ts_itest-base.cc:116] --------------
2024-05-02T14:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:01:08Z Disabled control of system clock
I20240502 14:01:08.140754 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:41117
--webserver_interface=127.16.217.62
--webserver_port=0
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.217.62:41117
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 14:01:08.241061 28853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:08.241266 28853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:08.241300 28853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:08.243970 28853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:08.244040 28853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:01:08.244062 28853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:08.244079 28853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:01:08.244102 28853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:01:08.247435 28853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.217.62:41117
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.217.62:41117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.217.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:08.248322 28853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:08.249388 28853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.256208 28859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.256217 28861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.256402 28858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.257004 28853 server_base.cc:1034] running on GCE node
I20240502 14:01:08.257467 28853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:08.258301 28853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:08.259454 28853 hybrid_clock.cc:648] HybridClock initialized: now 1714658468259437 us; error 36 us; skew 500 ppm
I20240502 14:01:08.266530 28853 webserver.cc:460] Webserver started at http://127.16.217.62:45855/ using document root <none> and password file <none>
I20240502 14:01:08.267042 28853 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.267113 28853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.267292 28853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.268891 28853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "db037a839b724e0babeaf39df2515fc8"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.269316 28853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "db037a839b724e0babeaf39df2515fc8"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.272379 28853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:01:08.274375 28867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.275359 28853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:08.275461 28853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
uuid: "db037a839b724e0babeaf39df2515fc8"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.275532 28853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.296012 28853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.296624 28853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:08.296774 28853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.302804 28853 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.62:41117
I20240502 14:01:08.302814 28919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.62:41117 every 8 connection(s)
I20240502 14:01:08.303678 28853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:01:08.305522 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 28853
I20240502 14:01:08.305682 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:01:08.306468 28920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:08.312364 28920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8: Bootstrap starting.
I20240502 14:01:08.314388 28920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:08.315167 28920 log.cc:824] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:08.316670 28920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8: No bootstrap required, opened a new log
I20240502 14:01:08.319094 28920 raft_consensus.cc:348] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } }
I20240502 14:01:08.319254 28920 raft_consensus.cc:374] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:08.319291 28920 raft_consensus.cc:729] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db037a839b724e0babeaf39df2515fc8, State: Initialized, Role: FOLLOWER
I20240502 14:01:08.319624 28920 consensus_queue.cc:260] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } }
I20240502 14:01:08.319720 28920 raft_consensus.cc:388] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:01:08.319761 28920 raft_consensus.cc:482] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:01:08.319828 28920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:08.320508 28920 raft_consensus.cc:504] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } }
I20240502 14:01:08.320770 28920 leader_election.cc:302] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db037a839b724e0babeaf39df2515fc8; no voters:
I20240502 14:01:08.320974 28920 leader_election.cc:288] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:01:08.321067 28926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:08.321578 28926 raft_consensus.cc:686] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [term 1 LEADER]: Becoming Leader. State: Replica: db037a839b724e0babeaf39df2515fc8, State: Running, Role: LEADER
I20240502 14:01:08.321842 28926 consensus_queue.cc:237] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } }
I20240502 14:01:08.321945 28920 sys_catalog.cc:564] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:01:08.323194 28928 sys_catalog.cc:455] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db037a839b724e0babeaf39df2515fc8. Latest consensus state: current_term: 1 leader_uuid: "db037a839b724e0babeaf39df2515fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } } }
I20240502 14:01:08.323215 28927 sys_catalog.cc:455] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db037a839b724e0babeaf39df2515fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db037a839b724e0babeaf39df2515fc8" member_type: VOTER last_known_addr { host: "127.16.217.62" port: 41117 } } }
I20240502 14:01:08.323292 28928 sys_catalog.cc:458] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [sys.catalog]: This master's current role is: LEADER
I20240502 14:01:08.323299 28927 sys_catalog.cc:458] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8 [sys.catalog]: This master's current role is: LEADER
I20240502 14:01:08.323632 28936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:01:08.326453 28936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:01:08.331422 28936 catalog_manager.cc:1347] Generated new cluster ID: 3c2f80cbaded46e58a3eca63935ddb25
I20240502 14:01:08.331475 28936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:01:08.345701 28936 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:01:08.346518 28936 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:01:08.359202 28936 catalog_manager.cc:5857] T 00000000000000000000000000000000 P db037a839b724e0babeaf39df2515fc8: Generated new TSK 0
I20240502 14:01:08.359841 28936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:01:08.367916 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.1:0
--local_ip_for_outbound_sockets=127.16.217.1
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:01:08.480263 28945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:08.480479 28945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:08.480541 28945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:08.483152 28945 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:08.483213 28945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:08.483299 28945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.1
I20240502 14:01:08.486570 28945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.217.1
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.28945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.1
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:08.487329 28945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:08.488482 28945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.495901 28951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.495901 28953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.496083 28950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.496587 28945 server_base.cc:1034] running on GCE node
I20240502 14:01:08.496922 28945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:08.497418 28945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:08.498572 28945 hybrid_clock.cc:648] HybridClock initialized: now 1714658468498545 us; error 38 us; skew 500 ppm
I20240502 14:01:08.500454 28945 webserver.cc:460] Webserver started at http://127.16.217.1:43483/ using document root <none> and password file <none>
I20240502 14:01:08.500963 28945 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.501032 28945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.501222 28945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.502714 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a7221b90d0aa4770a8ce5522103a96f4"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.503180 28945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a7221b90d0aa4770a8ce5522103a96f4"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.506330 28945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:01:08.508301 28959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.509224 28945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:08.509327 28945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a7221b90d0aa4770a8ce5522103a96f4"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.509404 28945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.528663 28945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.529304 28945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:08.529461 28945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.529983 28945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.530843 28945 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.530891 28945 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.530952 28945 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.530979 28945 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.538970 28945 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.1:37835
I20240502 14:01:08.539014 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.1:37835 every 8 connection(s)
I20240502 14:01:08.539878 28945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:01:08.544780 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 28945
I20240502 14:01:08.544898 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:01:08.547010 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.2:0
--local_ip_for_outbound_sockets=127.16.217.2
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:08.548636 29075 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:08.548868 29075 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.549276 29075 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:08.550558 28884 ts_manager.cc:194] Registered new tserver with Master: a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:08.551674 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.1:34213
W20240502 14:01:08.649569 29078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:08.649793 29078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:08.649868 29078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:08.652693 29078 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:08.652750 29078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:08.652835 29078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.2
I20240502 14:01:08.656174 29078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.217.2
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.2
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:08.656999 29078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:08.658027 29078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.665535 29086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.666110 29078 server_base.cc:1034] running on GCE node
W20240502 14:01:08.666203 29084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.666273 29083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.666632 29078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:08.667116 29078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:08.668256 29078 hybrid_clock.cc:648] HybridClock initialized: now 1714658468668244 us; error 32 us; skew 500 ppm
I20240502 14:01:08.669977 29078 webserver.cc:460] Webserver started at http://127.16.217.2:39563/ using document root <none> and password file <none>
I20240502 14:01:08.670522 29078 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.670583 29078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.670749 29078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.672348 29078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.672816 29078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.675949 29078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 14:01:08.677911 29092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.678824 29078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:08.678926 29078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.678997 29078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.694842 29078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.695326 29078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:08.695444 29078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.695852 29078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.696624 29078 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.696667 29078 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.696700 29078 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.696720 29078 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.703886 29078 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.2:35841
I20240502 14:01:08.703956 29206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.2:35841 every 8 connection(s)
I20240502 14:01:08.704785 29078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:01:08.712410 29207 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:08.712615 29207 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.713021 29207 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:08.713429 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29078
I20240502 14:01:08.713542 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:01:08.713933 28884 ts_manager.cc:194] Registered new tserver with Master: bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:08.714536 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.2:52837
I20240502 14:01:08.715844 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.3:0
--local_ip_for_outbound_sockets=127.16.217.3
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:01:08.814016 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:08.814231 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:08.814285 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:08.817140 29210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:08.817198 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:08.817288 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.3
I20240502 14:01:08.820461 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.217.3
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.3
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:08.821276 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:08.822384 29210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:08.830046 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.830140 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:08.830261 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:08.830840 29210 server_base.cc:1034] running on GCE node
I20240502 14:01:08.831179 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:08.831673 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:08.832805 29210 hybrid_clock.cc:648] HybridClock initialized: now 1714658468832785 us; error 32 us; skew 500 ppm
I20240502 14:01:08.834518 29210 webserver.cc:460] Webserver started at http://127.16.217.3:40349/ using document root <none> and password file <none>
I20240502 14:01:08.835037 29210 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:08.835095 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:08.835274 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:08.836829 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cd692a8958c241fca3aec5cca3a0ded5"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.837250 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cd692a8958c241fca3aec5cca3a0ded5"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.840183 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:01:08.842113 29224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.842991 29210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:08.843086 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cd692a8958c241fca3aec5cca3a0ded5"
format_stamp: "Formatted at 2024-05-02 14:01:08 on dist-test-slave-7msf"
I20240502 14:01:08.843163 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:08.879055 29210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:08.879663 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:08.879817 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:08.880388 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:08.881254 29210 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:08.881304 29210 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.881359 29210 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:08.881383 29210 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:08.888950 29210 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.3:41345
I20240502 14:01:08.888970 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.3:41345 every 8 connection(s)
I20240502 14:01:08.889770 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:01:08.891436 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29210
I20240502 14:01:08.891561 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:01:08.893741 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.4:0
--local_ip_for_outbound_sockets=127.16.217.4
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:08.898439 29339 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:08.898623 29339 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:08.898995 29339 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:08.899926 28883 ts_manager.cc:194] Registered new tserver with Master: cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:08.900710 28883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.3:51561
W20240502 14:01:08.994634 29342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:08.994817 29342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:08.994856 29342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:08.997547 29342 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:08.997596 29342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:08.997653 29342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.4
I20240502 14:01:09.000622 29342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.217.4
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.4
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:09.001299 29342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:09.002259 29342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:09.009467 29348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.009522 29350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.009650 29347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:09.010257 29342 server_base.cc:1034] running on GCE node
I20240502 14:01:09.010566 29342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:09.011022 29342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:09.012176 29342 hybrid_clock.cc:648] HybridClock initialized: now 1714658469012156 us; error 35 us; skew 500 ppm
I20240502 14:01:09.014089 29342 webserver.cc:460] Webserver started at http://127.16.217.4:38089/ using document root <none> and password file <none>
I20240502 14:01:09.014608 29342 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:09.014662 29342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:09.014854 29342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:09.016466 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "edce308627b14fa6968d90ec77d51aec"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.016906 29342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "edce308627b14fa6968d90ec77d51aec"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.019819 29342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:01:09.021775 29356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.022655 29342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:09.022774 29342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "edce308627b14fa6968d90ec77d51aec"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.022850 29342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:09.060374 29342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:09.060947 29342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:09.061095 29342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:09.061581 29342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:09.062412 29342 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:09.062456 29342 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.062541 29342 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:09.062585 29342 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.070191 29342 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.4:45347
I20240502 14:01:09.070220 29475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.4:45347 every 8 connection(s)
I20240502 14:01:09.071020 29342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:01:09.078812 29476 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:09.078999 29476 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:09.079373 29476 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:09.079814 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29342
I20240502 14:01:09.079934 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:01:09.080348 28883 ts_manager.cc:194] Registered new tserver with Master: edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:09.080863 28883 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.4:59143
I20240502 14:01:09.081828 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:0
--local_ip_for_outbound_sockets=127.16.217.5
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:01:09.179656 29479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:09.179841 29479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:09.179884 29479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:09.182498 29479 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:09.182564 29479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:09.182653 29479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 14:01:09.185817 29479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:09.186517 29479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:09.187470 29479 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:09.194721 29487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.194792 29485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.195135 29484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:09.195669 29479 server_base.cc:1034] running on GCE node
I20240502 14:01:09.195950 29479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:09.196450 29479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:09.197594 29479 hybrid_clock.cc:648] HybridClock initialized: now 1714658469197580 us; error 27 us; skew 500 ppm
I20240502 14:01:09.199234 29479 webserver.cc:460] Webserver started at http://127.16.217.5:39135/ using document root <none> and password file <none>
I20240502 14:01:09.199717 29479 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:09.199765 29479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:09.199904 29479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:09.201458 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "22dbbf207ba0440392e71a10c2ca78ca"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.201855 29479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "22dbbf207ba0440392e71a10c2ca78ca"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.204946 29479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:01:09.206936 29493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.207855 29479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:09.207940 29479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "22dbbf207ba0440392e71a10c2ca78ca"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.208009 29479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:09.221527 29479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:09.222002 29479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:09.222118 29479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:09.222543 29479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:09.223279 29479 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:09.223326 29479 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.223381 29479 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:09.223395 29479 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.230639 29479 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:45769
I20240502 14:01:09.230664 29609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:45769 every 8 connection(s)
I20240502 14:01:09.231515 29479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:01:09.237040 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29479
I20240502 14:01:09.237186 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:01:09.239374 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:0
--local_ip_for_outbound_sockets=127.16.217.6
--webserver_interface=127.16.217.6
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:09.240010 29610 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:09.240223 29610 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:09.240721 29610 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:09.241626 28884 ts_manager.cc:194] Registered new tserver with Master: 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:09.242100 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:58569
W20240502 14:01:09.337165 29613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:09.337365 29613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:09.337428 29613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:09.339946 29613 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:09.340008 29613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:09.340090 29613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 14:01:09.343093 29613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:09.343871 29613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:09.344965 29613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:09.352172 29619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.352233 29620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.352385 29622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:09.352885 29613 server_base.cc:1034] running on GCE node
I20240502 14:01:09.353165 29613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:09.353616 29613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:09.354777 29613 hybrid_clock.cc:648] HybridClock initialized: now 1714658469354763 us; error 24 us; skew 500 ppm
I20240502 14:01:09.356415 29613 webserver.cc:460] Webserver started at http://127.16.217.6:34871/ using document root <none> and password file <none>
I20240502 14:01:09.356930 29613 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:09.356981 29613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:09.357133 29613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:09.358654 29613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "12348ca540074b32912fdd6333484e0b"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.359076 29613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "12348ca540074b32912fdd6333484e0b"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.362021 29613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 14:01:09.363880 29628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.364815 29613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:09.364912 29613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "12348ca540074b32912fdd6333484e0b"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.364987 29613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:09.380860 29613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:09.381397 29613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:09.381547 29613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:09.382051 29613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:09.382860 29613 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:09.382915 29613 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.382969 29613 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:09.383004 29613 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.390321 29613 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:38737
I20240502 14:01:09.390347 29742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:38737 every 8 connection(s)
I20240502 14:01:09.391176 29613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:01:09.394834 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29613
I20240502 14:01:09.394973 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:01:09.397300 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:0
--local_ip_for_outbound_sockets=127.16.217.7
--webserver_interface=127.16.217.7
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:09.399765 29743 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:09.399950 29743 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:09.400358 29743 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:09.401199 28884 ts_manager.cc:194] Registered new tserver with Master: 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:01:09.401679 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:52663
W20240502 14:01:09.496873 29747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:09.497064 29747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:09.497125 29747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:09.499753 29747 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:09.499816 29747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:09.499903 29747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 14:01:09.502925 29747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:09.503652 29747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:09.504679 29747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:09.511936 29752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.512085 29755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.512169 29753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:09.512671 29747 server_base.cc:1034] running on GCE node
I20240502 14:01:09.513033 29747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:09.513562 29747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:09.514724 29747 hybrid_clock.cc:648] HybridClock initialized: now 1714658469514699 us; error 37 us; skew 500 ppm
I20240502 14:01:09.516708 29747 webserver.cc:460] Webserver started at http://127.16.217.7:44249/ using document root <none> and password file <none>
I20240502 14:01:09.517249 29747 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:09.517311 29747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:09.517488 29747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:09.519196 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "33ebf308f6f24e6e8768ef5f14ddc785"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.519663 29747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "33ebf308f6f24e6e8768ef5f14ddc785"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.523233 29747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240502 14:01:09.525408 29761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.526343 29747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:09.526465 29747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "33ebf308f6f24e6e8768ef5f14ddc785"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.526537 29747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:09.554258 29075 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:09.555303 29747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:09.555892 29747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:09.556023 29747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:09.556499 29747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:09.557332 29747 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:09.557374 29747 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.557410 29747 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:09.557430 29747 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.564993 29747 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:40969
I20240502 14:01:09.565024 29874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:40969 every 8 connection(s)
I20240502 14:01:09.565863 29747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:01:09.573143 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29747
I20240502 14:01:09.573263 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:01:09.574383 29875 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:09.574754 29875 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:09.575688 29875 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:09.576484 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.8:0
--local_ip_for_outbound_sockets=127.16.217.8
--webserver_interface=127.16.217.8
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:09.577888 28884 ts_manager.cc:194] Registered new tserver with Master: 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:09.579715 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:53227
W20240502 14:01:09.680297 29878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:09.680522 29878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:09.680567 29878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:09.683503 29878 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:09.683573 29878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:09.683660 29878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.8
I20240502 14:01:09.686964 29878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.217.8
--webserver_port=0
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.29878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.8
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:09.687845 29878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:09.688935 29878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:09.696260 29887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.696406 29884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:09.696478 29885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:09.696688 29878 server_base.cc:1034] running on GCE node
I20240502 14:01:09.696992 29878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:09.697458 29878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:09.698591 29878 hybrid_clock.cc:648] HybridClock initialized: now 1714658469698577 us; error 31 us; skew 500 ppm
I20240502 14:01:09.700227 29878 webserver.cc:460] Webserver started at http://127.16.217.8:40059/ using document root <none> and password file <none>
I20240502 14:01:09.700700 29878 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:09.700768 29878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:09.700910 29878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:01:09.702361 29878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c59dc7abff9749abab221ce409d49f59"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.702761 29878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c59dc7abff9749abab221ce409d49f59"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.705912 29878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 14:01:09.707784 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.708752 29878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:01:09.708839 29878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c59dc7abff9749abab221ce409d49f59"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:09.708917 29878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:09.716445 29207 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:09.723542 29878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:09.724094 29878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:09.724236 29878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:09.724725 29878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:09.725517 29878 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:09.725558 29878 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.725593 29878 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:09.725615 29878 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:09.733493 29878 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.8:38573
I20240502 14:01:09.733564 30007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.8:38573 every 8 connection(s)
I20240502 14:01:09.734391 29878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:01:09.741761 30008 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:09.741943 30008 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:09.742318 30008 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:09.743145 28884 ts_manager.cc:194] Registered new tserver with Master: c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:01:09.743659 28884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.8:36991
I20240502 14:01:09.744489 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 29878
I20240502 14:01:09.744596 17252 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:01:09.746644 17252 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:01:09.754597 17252 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:01:09.754662 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 29479
I20240502 14:01:09.760888 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 29613
I20240502 14:01:09.766072 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 29747
I20240502 14:01:09.771178 17252 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhNHehp/build/debug/bin/kudu with pid 29878
I20240502 14:01:09.902464 29339 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:10.082738 29476 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:13.527865 28884 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:13.547880 29405 tablet_service.cc:1467] Processing CreateTablet for tablet 372e9cc5aaed4b80b3b451efdb3e94fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.548022 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 372e9cc5aaed4b80b3b451efdb3e94fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.548859 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372e9cc5aaed4b80b3b451efdb3e94fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.548897 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372e9cc5aaed4b80b3b451efdb3e94fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.549481 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1b37b67e104154b6755c1762f8f6ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.550217 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.549495 29008 tablet_service.cc:1467] Processing CreateTablet for tablet 372e9cc5aaed4b80b3b451efdb3e94fc (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.550657 29006 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0261898a445949f3cf21d2efb599f (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.550881 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0261898a445949f3cf21d2efb599f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.552553 29404 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0261898a445949f3cf21d2efb599f (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.552789 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0261898a445949f3cf21d2efb599f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.554440 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1b37b67e104154b6755c1762f8f6ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.554687 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.555923 30028 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:13.556526 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372e9cc5aaed4b80b3b451efdb3e94fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.557360 30029 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:13.558229 30028 tablet_bootstrap.cc:654] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.558844 30029 tablet_bootstrap.cc:654] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.559015 30028 log.cc:824] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:13.559540 30029 log.cc:824] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:13.560331 30030 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:13.561338 30028 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:13.561558 30028 ts_tablet_manager.cc:1367] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:01:13.562038 30029 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:13.562268 30029 ts_tablet_manager.cc:1367] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:01:13.563838 30030 tablet_bootstrap.cc:654] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.565359 30030 log.cc:824] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:13.565330 29140 tablet_service.cc:1467] Processing CreateTablet for tablet d9e0261898a445949f3cf21d2efb599f (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.565685 30028 raft_consensus.cc:348] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.565883 30028 raft_consensus.cc:374] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.565930 30028 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.566105 30029 raft_consensus.cc:348] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.566289 30029 raft_consensus.cc:374] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.566288 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0261898a445949f3cf21d2efb599f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.566330 30029 raft_consensus.cc:729] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.566395 30028 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.566757 30029 consensus_queue.cc:260] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.567093 30030 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:13.567215 30028 ts_tablet_manager.cc:1398] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:01:13.567242 30030 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:01:13.567544 30028 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:13.567595 30029 ts_tablet_manager.cc:1398] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:01:13.567945 30029 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:13.568760 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1b37b67e104154b6755c1762f8f6ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.568893 30029 tablet_bootstrap.cc:654] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.569023 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.569777 30028 tablet_bootstrap.cc:654] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.570574 30030 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.570870 30028 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:13.570947 30028 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:13.571453 30028 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.571560 30028 raft_consensus.cc:374] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.571589 30028 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.571708 30028 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.572037 30028 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:13.573045 30030 raft_consensus.cc:374] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.573153 30030 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.573741 30030 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.574291 30040 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
W20240502 14:01:13.574903 29477 tablet.cc:2369] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:01:13.575155 30029 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:13.575223 30029 ts_tablet_manager.cc:1367] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240502 14:01:13.575740 30029 raft_consensus.cc:348] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.575832 30029 raft_consensus.cc:374] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.575872 30029 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.575914 30030 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:01:13.576017 30029 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.576318 30029 ts_tablet_manager.cc:1398] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:13.576493 30030 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:13.577564 30030 tablet_bootstrap.cc:654] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.578238 30040 tablet_bootstrap.cc:654] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.578763 30030 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:13.578891 30030 ts_tablet_manager.cc:1367] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:13.579052 30040 log.cc:824] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:13.579378 30030 raft_consensus.cc:348] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.579499 30030 raft_consensus.cc:374] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.579552 30030 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.579684 30030 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.579921 30030 ts_tablet_manager.cc:1398] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:13.580118 30030 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:13.581106 30040 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:13.581418 30040 ts_tablet_manager.cc:1367] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:01:13.582403 30030 tablet_bootstrap.cc:654] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.584040 30030 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:13.584107 30030 ts_tablet_manager.cc:1367] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:01:13.584470 30030 raft_consensus.cc:348] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.584547 30030 raft_consensus.cc:374] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.584568 30030 raft_consensus.cc:729] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.584684 30030 consensus_queue.cc:260] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.584897 30030 ts_tablet_manager.cc:1398] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:13.585697 30040 raft_consensus.cc:348] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.585876 30040 raft_consensus.cc:374] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.585938 30040 raft_consensus.cc:729] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.586311 30040 consensus_queue.cc:260] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.587810 30040 ts_tablet_manager.cc:1398] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:01:13.588006 30040 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:13.588847 30040 tablet_bootstrap.cc:654] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.589908 30040 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:13.589977 30040 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:13.590374 30040 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.590453 30040 raft_consensus.cc:374] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.590483 30040 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.590595 30040 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.590821 30040 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:13.592017 30041 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:13.592154 30041 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.593107 30041 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:13.596624 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:13.596649 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:13.596858 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:13.596866 29426 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:13.597237 28962 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:13.597465 30041 raft_consensus.cc:2781] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:13.597527 30041 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:13.597553 30041 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.598361 30041 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.598687 30041 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:13.598945 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:13.598999 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:13.599054 29426 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.599104 29293 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.599898 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:13.599902 29426 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:13.600157 28962 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:13.600348 30041 raft_consensus.cc:2781] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:13.600515 30041 raft_consensus.cc:686] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:13.600777 30041 consensus_queue.cc:237] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:13.603356 28884 catalog_manager.cc:5484] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
W20240502 14:01:13.643976 29340 tablet.cc:2369] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:01:13.657804 30039 raft_consensus.cc:482] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:13.657938 30039 raft_consensus.cc:504] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.658947 30039 leader_election.cc:288] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:13.662503 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1b37b67e104154b6755c1762f8f6ff" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:13.662609 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1b37b67e104154b6755c1762f8f6ff" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:13.662812 29161 raft_consensus.cc:2454] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:13.662825 29028 raft_consensus.cc:2454] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:13.663223 29227 leader_election.cc:302] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:13.663471 30039 raft_consensus.cc:2781] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:13.663527 30039 raft_consensus.cc:482] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:13.663551 30039 raft_consensus.cc:3037] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.664896 30039 raft_consensus.cc:504] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.665251 30039 leader_election.cc:288] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:13.665521 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1b37b67e104154b6755c1762f8f6ff" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:13.665617 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1b37b67e104154b6755c1762f8f6ff" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:13.665643 29161 raft_consensus.cc:3037] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.665730 29028 raft_consensus.cc:3037] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.666450 29161 raft_consensus.cc:2454] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:13.666500 29028 raft_consensus.cc:2454] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:13.666738 29226 leader_election.cc:302] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:13.666898 30039 raft_consensus.cc:2781] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:13.666986 30039 raft_consensus.cc:686] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:13.667261 30039 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:13.669656 28884 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:01:13.707044 30041 raft_consensus.cc:482] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:13.707170 30041 raft_consensus.cc:504] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.707651 30041 leader_election.cc:288] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:13.708258 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0261898a445949f3cf21d2efb599f" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:13.708475 29426 raft_consensus.cc:2454] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:13.708781 28960 leader_election.cc:302] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:13.708959 30041 raft_consensus.cc:2781] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:13.709013 30041 raft_consensus.cc:482] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:13.709064 30041 raft_consensus.cc:3037] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:01:13.709394 29208 tablet.cc:2369] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:01:13.710037 30041 raft_consensus.cc:504] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.710369 30041 leader_election.cc:288] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:13.710698 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0261898a445949f3cf21d2efb599f" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:13.710796 29426 raft_consensus.cc:3037] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.711707 29426 raft_consensus.cc:2454] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:13.711758 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0261898a445949f3cf21d2efb599f" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:13.711819 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e0261898a445949f3cf21d2efb599f" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:13.711865 29161 raft_consensus.cc:2454] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:13.711985 28960 leader_election.cc:302] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:13.712123 30041 raft_consensus.cc:2781] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:13.712195 30041 raft_consensus.cc:686] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:13.712441 30041 consensus_queue.cc:237] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.714026 28884 catalog_manager.cc:5484] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:01:13.794446 29076 tablet.cc:2369] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:01:13.830971 30055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:13.831183 30055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:13.834096 30055 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:01:13.923138 29160 raft_consensus.cc:3037] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:13.923671 29426 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.924167 29425 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.924335 29160 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.924666 29293 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.925335 30049 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.925776 30041 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.926129 30041 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.927800 29028 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.928690 30039 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.929973 30049 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.933971 29161 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:13.937798 30035 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:13.944600 30089 mvcc.cc:205] Tried to move back new op lower bound from 7023241109183971328 to 7023241107870863360. Current Snapshot: MvccSnapshot[applied={T|T < 7023241109183971328}]
I20240502 14:01:13.947161 30092 mvcc.cc:205] Tried to move back new op lower bound from 7023241109199282176 to 7023241107870863360. Current Snapshot: MvccSnapshot[applied={T|T < 7023241109183971328}]
INSERT report
rows total: 2000
time total: 99.027 ms
time per row: 0.0495135 ms
I20240502 14:01:13.967809 30091 mvcc.cc:205] Tried to move back new op lower bound from 7023241109200838656 to 7023241108143398912. Current Snapshot: MvccSnapshot[applied={T|T < 7023241109200838656}]
I20240502 14:01:13.979552 28884 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:13.985026 29141 tablet_service.cc:1467] Processing CreateTablet for tablet cfd44eeeec934413a5001673430e6500 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.985214 29272 tablet_service.cc:1467] Processing CreateTablet for tablet cfd44eeeec934413a5001673430e6500 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.985369 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd44eeeec934413a5001673430e6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.985515 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd44eeeec934413a5001673430e6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.986135 29006 tablet_service.cc:1467] Processing CreateTablet for tablet e6787832a36648aca61dd26a363089d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.986403 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6787832a36648aca61dd26a363089d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.987100 30028 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:13.987552 29405 tablet_service.cc:1467] Processing CreateTablet for tablet cfd44eeeec934413a5001673430e6500 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.987637 29273 tablet_service.cc:1467] Processing CreateTablet for tablet e6787832a36648aca61dd26a363089d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.987790 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd44eeeec934413a5001673430e6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.987854 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6787832a36648aca61dd26a363089d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.987900 30028 tablet_bootstrap.cc:654] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.989269 30028 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:13.989362 30028 ts_tablet_manager.cc:1367] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:13.989709 29404 tablet_service.cc:1467] Processing CreateTablet for tablet ff65ce0568514e89894ec845fbda31f3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.989830 30028 raft_consensus.cc:348] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.989912 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.989930 30028 raft_consensus.cc:374] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.989964 30028 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.989948 29272 tablet_service.cc:1467] Processing CreateTablet for tablet ff65ce0568514e89894ec845fbda31f3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.990070 30028 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.990360 30028 ts_tablet_manager.cc:1398] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:13.990137 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.990797 30029 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:13.991556 30029 tablet_bootstrap.cc:654] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.992086 30030 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:13.992482 30029 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:13.992548 30029 ts_tablet_manager.cc:1367] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:13.992609 30028 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:13.992877 30030 tablet_bootstrap.cc:654] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.992869 30029 raft_consensus.cc:348] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.992941 30029 raft_consensus.cc:374] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.992974 30029 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.993096 30029 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.993294 30029 ts_tablet_manager.cc:1398] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:13.993422 30028 tablet_bootstrap.cc:654] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.993429 30029 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:13.993813 30040 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:13.994158 30029 tablet_bootstrap.cc:654] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.994457 30030 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:13.994478 30040 tablet_bootstrap.cc:654] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.994527 30030 ts_tablet_manager.cc:1367] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:13.994944 30030 raft_consensus.cc:348] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:13.995041 30030 raft_consensus.cc:374] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.995072 30030 raft_consensus.cc:729] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.995196 30029 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:13.995208 30030 consensus_queue.cc:260] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:13.995258 30029 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:13.995458 30030 ts_tablet_manager.cc:1398] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:13.995620 30029 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.995698 30029 raft_consensus.cc:374] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.995721 30029 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.995829 30029 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.996104 30029 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:13.996155 29141 tablet_service.cc:1467] Processing CreateTablet for tablet ff65ce0568514e89894ec845fbda31f3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.996423 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.996773 30040 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:13.996843 30040 ts_tablet_manager.cc:1367] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:13.997217 30040 raft_consensus.cc:348] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.997308 30040 raft_consensus.cc:374] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:13.997337 30040 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:13.997454 30040 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:13.997687 30040 ts_tablet_manager.cc:1398] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:13.997887 30040 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:13.998090 29140 tablet_service.cc:1467] Processing CreateTablet for tablet e6787832a36648aca61dd26a363089d5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:13.998291 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6787832a36648aca61dd26a363089d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:13.998610 30040 tablet_bootstrap.cc:654] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:13.999688 30040 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:13.999742 30040 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:14.000069 30040 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.000160 30040 raft_consensus.cc:374] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.000195 30040 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.000324 30040 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.000507 30040 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.000640 30040 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:14.000896 30028 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:14.000988 30028 ts_tablet_manager.cc:1367] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:01:14.001428 30028 raft_consensus.cc:348] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.001543 30028 raft_consensus.cc:374] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.001582 30040 tablet_bootstrap.cc:654] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.001585 30028 raft_consensus.cc:729] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.001698 30028 consensus_queue.cc:260] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.001946 30028 ts_tablet_manager.cc:1398] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.002099 30028 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:14.002801 30028 tablet_bootstrap.cc:654] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.003814 30028 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:14.003891 30028 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:14.004328 30040 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:14.004308 30028 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.004410 30040 ts_tablet_manager.cc:1367] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:14.004410 30028 raft_consensus.cc:374] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.004446 30028 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.004611 30028 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.004818 30040 raft_consensus.cc:348] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.004914 30040 raft_consensus.cc:374] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.004923 30028 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.004957 30040 raft_consensus.cc:729] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.005079 30040 consensus_queue.cc:260] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.005299 30040 ts_tablet_manager.cc:1398] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:14.047919 30045 raft_consensus.cc:482] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.048069 30045 raft_consensus.cc:504] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.049130 30045 leader_election.cc:288] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.053244 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:14.053375 29293 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.053604 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:14.053728 29426 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.053710 29095 leader_election.cc:302] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:14.053977 30045 raft_consensus.cc:2781] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.054041 30045 raft_consensus.cc:482] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.054076 30045 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.054924 30045 raft_consensus.cc:504] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.055217 30045 leader_election.cc:288] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.055481 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:14.055547 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:14.055613 29293 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.055661 29426 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.056375 29293 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.056430 29426 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.056630 29095 leader_election.cc:302] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:14.056779 30045 raft_consensus.cc:2781] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.057004 30045 raft_consensus.cc:686] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:14.057263 30045 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.059041 28877 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:14.076200 30045 raft_consensus.cc:482] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.076332 30045 raft_consensus.cc:504] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.076797 30045 leader_election.cc:288] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:14.077435 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6787832a36648aca61dd26a363089d5" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:14.077569 29293 raft_consensus.cc:2454] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.077828 29095 leader_election.cc:302] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:14.078002 30045 raft_consensus.cc:2781] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.078043 30045 raft_consensus.cc:482] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.078080 30045 raft_consensus.cc:3037] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.079111 30045 raft_consensus.cc:504] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.079386 30045 leader_election.cc:288] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:14.079721 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6787832a36648aca61dd26a363089d5" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:14.079850 29293 raft_consensus.cc:3037] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.080629 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6787832a36648aca61dd26a363089d5" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:14.080772 29028 raft_consensus.cc:2454] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.080775 29293 raft_consensus.cc:2454] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.080972 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6787832a36648aca61dd26a363089d5" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:14.081060 29027 raft_consensus.cc:3037] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.081203 29095 leader_election.cc:302] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:14.081367 30045 raft_consensus.cc:2781] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.081509 30045 raft_consensus.cc:686] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:14.081625 30045 consensus_queue.cc:237] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:14.081913 29027 raft_consensus.cc:2454] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.083107 28877 catalog_manager.cc:5484] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:01:14.160367 30035 raft_consensus.cc:482] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.160501 30035 raft_consensus.cc:504] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.161011 30035 leader_election.cc:288] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.161417 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff65ce0568514e89894ec845fbda31f3" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:14.161564 29160 raft_consensus.cc:2454] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:14.161849 29226 leader_election.cc:302] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:14.162000 30035 raft_consensus.cc:2781] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.162065 30035 raft_consensus.cc:482] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.162108 30035 raft_consensus.cc:3037] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.163025 30035 raft_consensus.cc:504] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.163319 30035 leader_election.cc:288] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.163699 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff65ce0568514e89894ec845fbda31f3" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:14.163798 29160 raft_consensus.cc:3037] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.164299 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff65ce0568514e89894ec845fbda31f3" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:14.164338 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff65ce0568514e89894ec845fbda31f3" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:14.164594 29426 raft_consensus.cc:2454] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:14.164870 29160 raft_consensus.cc:2454] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:14.165113 29226 leader_election.cc:302] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters: edce308627b14fa6968d90ec77d51aec
I20240502 14:01:14.165257 30035 raft_consensus.cc:2781] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.165343 30035 raft_consensus.cc:686] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:14.165477 30035 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.166960 28877 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:01:14.285538 30131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:14.285763 30131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:14.288760 30131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:01:14.355891 29160 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.356355 30039 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.359138 29426 raft_consensus.cc:3037] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.360177 29426 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.360556 30035 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.370339 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.370599 29293 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.371122 30045 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.374250 29292 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.374609 29027 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:14.375048 30045 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.375339 30045 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.376083 30045 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:14.391889 30085 mvcc.cc:205] Tried to move back new op lower bound from 7023241111013765120 to 7023241109839663104. Current Snapshot: MvccSnapshot[applied={T|T < 7023241111013765120}]
INSERT report
rows total: 2000
time total: 94.0193 ms
time per row: 0.0470096 ms
I20240502 14:01:14.419618 28877 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:14.424806 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa944cd0b82429da02bd3dd35e0cce7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.424911 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa944cd0b82429da02bd3dd35e0cce7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.425105 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.425220 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.426074 29403 tablet_service.cc:1467] Processing CreateTablet for tablet 86cfc7d5709f414aac098ae3c05a48a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.426064 29272 tablet_service.cc:1467] Processing CreateTablet for tablet db68071f838a4e209f3db69bea6e54f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.426337 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.426340 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db68071f838a4e209f3db69bea6e54f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.427182 30040 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:14.427340 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa944cd0b82429da02bd3dd35e0cce7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.427520 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.428000 30040 tablet_bootstrap.cc:654] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.428424 30028 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:14.428532 29405 tablet_service.cc:1467] Processing CreateTablet for tablet db68071f838a4e209f3db69bea6e54f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.428601 30029 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:14.428714 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db68071f838a4e209f3db69bea6e54f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.429073 30040 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:14.429137 30040 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:14.429265 30028 tablet_bootstrap.cc:654] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.429571 30029 tablet_bootstrap.cc:654] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.429558 30040 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.429647 30040 raft_consensus.cc:374] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.429682 30040 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.429795 30040 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.430032 30040 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.430094 30030 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:14.430948 30029 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:14.430968 30030 tablet_bootstrap.cc:654] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.431025 30029 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:14.431391 30029 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.431469 30029 raft_consensus.cc:374] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.431502 30029 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.431622 30029 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.431876 30029 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.431979 30028 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:14.432052 30029 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:14.432078 30028 ts_tablet_manager.cc:1367] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:14.432528 30028 raft_consensus.cc:348] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.432554 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 86cfc7d5709f414aac098ae3c05a48a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.432627 30028 raft_consensus.cc:374] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.432662 30028 raft_consensus.cc:729] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.432754 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.432790 30028 consensus_queue.cc:260] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.432847 30029 tablet_bootstrap.cc:654] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.433041 30028 ts_tablet_manager.cc:1398] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.433209 29006 tablet_service.cc:1467] Processing CreateTablet for tablet db68071f838a4e209f3db69bea6e54f3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.433387 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db68071f838a4e209f3db69bea6e54f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.434147 30029 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:14.434214 30029 ts_tablet_manager.cc:1367] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:14.434631 30029 raft_consensus.cc:348] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.434747 30029 raft_consensus.cc:374] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.434774 30029 raft_consensus.cc:729] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.434916 30040 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:14.434916 30029 consensus_queue.cc:260] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.435106 29008 tablet_service.cc:1467] Processing CreateTablet for tablet 86cfc7d5709f414aac098ae3c05a48a3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:14.435220 30029 ts_tablet_manager.cc:1398] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:14.435330 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:14.435384 30029 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:14.435743 30040 tablet_bootstrap.cc:654] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.436237 30029 tablet_bootstrap.cc:654] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.436830 30040 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:14.436892 30040 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:14.437191 30029 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:14.437247 30029 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:14.437395 30040 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.437472 30040 raft_consensus.cc:374] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.437500 30040 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.437601 30029 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.437680 30029 raft_consensus.cc:374] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.437716 30029 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.437814 30040 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.437839 30029 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.438104 30040 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.438174 30029 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:14.438778 30030 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:14.438843 30030 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:01:14.439790 30030 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.439885 30030 raft_consensus.cc:374] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.439913 30030 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.440025 30030 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.440241 30030 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:14.440411 30030 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:14.441200 30030 tablet_bootstrap.cc:654] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.442034 30030 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:14.442111 30030 ts_tablet_manager.cc:1367] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:14.442412 30030 raft_consensus.cc:348] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.442482 30030 raft_consensus.cc:374] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.442500 30030 raft_consensus.cc:729] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.442608 30030 consensus_queue.cc:260] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.442842 30030 ts_tablet_manager.cc:1398] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:14.443032 30030 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:14.443679 30030 tablet_bootstrap.cc:654] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:14.444728 30030 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:14.444806 30030 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:14.445288 30030 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.445379 30030 raft_consensus.cc:374] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:14.445408 30030 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:14.445525 30030 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.445765 30030 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:14.496546 30166 raft_consensus.cc:482] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.496668 30166 raft_consensus.cc:504] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.497056 30166 leader_election.cc:288] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.497449 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa944cd0b82429da02bd3dd35e0cce7" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:14.497448 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa944cd0b82429da02bd3dd35e0cce7" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:14.497599 29426 raft_consensus.cc:2454] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.497596 29027 raft_consensus.cc:2454] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.497879 29093 leader_election.cc:302] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.498030 30166 raft_consensus.cc:2781] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.498083 30166 raft_consensus.cc:482] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.498126 30166 raft_consensus.cc:3037] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.498908 30166 raft_consensus.cc:504] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.499222 30166 leader_election.cc:288] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.499559 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa944cd0b82429da02bd3dd35e0cce7" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:14.499559 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa944cd0b82429da02bd3dd35e0cce7" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:14.499684 29027 raft_consensus.cc:3037] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.499711 29426 raft_consensus.cc:3037] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.500675 29027 raft_consensus.cc:2454] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.500675 29426 raft_consensus.cc:2454] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.500952 29093 leader_election.cc:302] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.501129 30166 raft_consensus.cc:2781] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.501204 30166 raft_consensus.cc:686] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:14.501332 30166 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.502810 28884 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:14.807726 30166 raft_consensus.cc:482] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.807868 30166 raft_consensus.cc:504] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.808255 30166 leader_election.cc:288] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.808621 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cfc7d5709f414aac098ae3c05a48a3" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:14.808641 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cfc7d5709f414aac098ae3c05a48a3" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:14.808737 29426 raft_consensus.cc:2454] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.808805 29027 raft_consensus.cc:2454] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:14.809019 29093 leader_election.cc:302] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.809218 30166 raft_consensus.cc:2781] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.809266 30166 raft_consensus.cc:482] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.809304 30166 raft_consensus.cc:3037] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.810093 30166 raft_consensus.cc:504] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.810416 30166 leader_election.cc:288] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:14.810667 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cfc7d5709f414aac098ae3c05a48a3" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:14.810760 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86cfc7d5709f414aac098ae3c05a48a3" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:14.810782 29027 raft_consensus.cc:3037] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.810853 29426 raft_consensus.cc:3037] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.811661 29027 raft_consensus.cc:2454] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.811677 29426 raft_consensus.cc:2454] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:14.811790 30088 raft_consensus.cc:482] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:14.811880 30088 raft_consensus.cc:504] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.811944 29093 leader_election.cc:302] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.812068 30166 raft_consensus.cc:2781] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.812140 30166 raft_consensus.cc:686] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:14.812268 30088 leader_election.cc:288] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:14.812383 30166 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:14.812528 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db68071f838a4e209f3db69bea6e54f3" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:14.812624 29426 raft_consensus.cc:2454] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:14.812780 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db68071f838a4e209f3db69bea6e54f3" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:14.812846 28960 leader_election.cc:302] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.812933 29292 raft_consensus.cc:2454] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:14.813365 30088 raft_consensus.cc:2781] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:14.813434 30088 raft_consensus.cc:482] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:14.813488 30088 raft_consensus.cc:3037] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.814152 28884 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:14.814335 30088 raft_consensus.cc:504] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.814718 30088 leader_election.cc:288] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:14.814985 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db68071f838a4e209f3db69bea6e54f3" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:14.815093 29426 raft_consensus.cc:3037] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.815057 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db68071f838a4e209f3db69bea6e54f3" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:14.815179 29292 raft_consensus.cc:3037] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:14.815953 29426 raft_consensus.cc:2454] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:14.816164 29292 raft_consensus.cc:2454] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:14.816202 28960 leader_election.cc:302] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:14.816447 30088 raft_consensus.cc:2781] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:14.816601 30088 raft_consensus.cc:686] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:14.816726 30088 consensus_queue.cc:237] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:14.818351 28884 catalog_manager.cc:5484] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
W20240502 14:01:14.991693 30180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:14.991919 30180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:14.994684 30180 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:01:15.021440 30166 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.024530 30166 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.080045 29426 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.080045 29292 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.080472 30178 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.080608 30088 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.085286 29027 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.086782 30166 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.088645 29425 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.090564 30045 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.048 ms
time per row: 0.0525238 ms
I20240502 14:01:15.141569 28883 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:15.146324 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4087c0e5443093e856fe676839b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.146368 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4087c0e5443093e856fe676839b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.146642 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4087c0e5443093e856fe676839b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.146654 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4087c0e5443093e856fe676839b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.146937 29008 tablet_service.cc:1467] Processing CreateTablet for tablet 252f4087c0e5443093e856fe676839b5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.147194 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4087c0e5443093e856fe676839b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.148602 30240 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:15.149091 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 0669ab13586a48ff9382fe1b80f47553 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.149139 29405 tablet_service.cc:1467] Processing CreateTablet for tablet 015b587864d54da8aee36c3afd968e86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.149339 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0669ab13586a48ff9382fe1b80f47553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.149514 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015b587864d54da8aee36c3afd968e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.149856 30240 tablet_bootstrap.cc:654] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.149964 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 015b587864d54da8aee36c3afd968e86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.150156 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015b587864d54da8aee36c3afd968e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.150399 29008 tablet_service.cc:1467] Processing CreateTablet for tablet 0669ab13586a48ff9382fe1b80f47553 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.150614 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0669ab13586a48ff9382fe1b80f47553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.151542 30244 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:15.151630 30242 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:15.151958 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 015b587864d54da8aee36c3afd968e86 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.152179 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015b587864d54da8aee36c3afd968e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.152518 30244 tablet_bootstrap.cc:654] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.152647 30242 tablet_bootstrap.cc:654] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.153566 30242 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:15.153569 30244 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:15.153649 30242 ts_tablet_manager.cc:1367] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.153654 30244 ts_tablet_manager.cc:1367] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.154049 30242 raft_consensus.cc:348] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.154156 30242 raft_consensus.cc:374] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.154114 30244 raft_consensus.cc:348] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.154196 30242 raft_consensus.cc:729] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.154208 30244 raft_consensus.cc:374] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.154240 30244 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.154309 30242 consensus_queue.cc:260] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.154368 30244 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.154603 30242 ts_tablet_manager.cc:1398] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.154623 30244 ts_tablet_manager.cc:1398] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.154770 30242 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:15.155082 30240 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:15.155171 30240 ts_tablet_manager.cc:1367] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:01:15.155544 30242 tablet_bootstrap.cc:654] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.155526 30240 raft_consensus.cc:348] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.155640 30240 raft_consensus.cc:374] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.155706 30240 raft_consensus.cc:729] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.155822 30240 consensus_queue.cc:260] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.156077 30240 ts_tablet_manager.cc:1398] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.156242 30240 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:15.156857 30244 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:15.156924 30240 tablet_bootstrap.cc:654] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.157634 30244 tablet_bootstrap.cc:654] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.157737 30240 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:15.157703 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 0669ab13586a48ff9382fe1b80f47553 (DEFAULT_TABLE table=rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.157790 30240 ts_tablet_manager.cc:1367] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:15.157892 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0669ab13586a48ff9382fe1b80f47553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.158186 30240 raft_consensus.cc:348] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.158265 30240 raft_consensus.cc:374] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.158303 30240 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.158387 30240 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.158576 30240 ts_tablet_manager.cc:1398] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:15.159328 30242 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:15.159400 30242 ts_tablet_manager.cc:1367] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:15.159698 30240 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:15.159775 30242 raft_consensus.cc:348] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.159868 30241 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:15.159879 30242 raft_consensus.cc:374] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.159914 30242 raft_consensus.cc:729] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.160033 30242 consensus_queue.cc:260] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.160425 30242 ts_tablet_manager.cc:1398] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.160462 30240 tablet_bootstrap.cc:654] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.160764 30241 tablet_bootstrap.cc:654] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.161077 30244 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:15.161139 30244 ts_tablet_manager.cc:1367] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:15.161676 30240 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:15.161684 30241 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:15.161756 30240 ts_tablet_manager.cc:1367] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:01:15.161756 30241 ts_tablet_manager.cc:1367] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.161872 30244 raft_consensus.cc:348] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.161994 30244 raft_consensus.cc:374] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.162031 30244 raft_consensus.cc:729] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.162147 30244 consensus_queue.cc:260] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.162147 30240 raft_consensus.cc:348] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.162163 30241 raft_consensus.cc:348] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.162227 30240 raft_consensus.cc:374] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.162237 30241 raft_consensus.cc:374] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.162257 30240 raft_consensus.cc:729] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.162266 30241 raft_consensus.cc:729] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.162359 30240 consensus_queue.cc:260] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.162365 30241 consensus_queue.cc:260] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.162391 30244 ts_tablet_manager.cc:1398] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.162595 30240 ts_tablet_manager.cc:1398] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.162606 30241 ts_tablet_manager.cc:1398] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.162743 30241 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:15.163492 30241 tablet_bootstrap.cc:654] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.164773 30241 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:15.164855 30241 ts_tablet_manager.cc:1367] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.165295 30241 raft_consensus.cc:348] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.165405 30241 raft_consensus.cc:374] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.165444 30241 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.165580 30241 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.165807 30241 ts_tablet_manager.cc:1398] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.226255 30166 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.226397 30166 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.226924 30166 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.227329 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:15.227337 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:15.227469 29292 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:15.227478 29425 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:15.227766 29093 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:15.227946 30166 raft_consensus.cc:2781] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.228015 30166 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.228067 30166 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.228920 30166 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.229209 30166 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.229503 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:15.229530 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:15.229593 29425 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.229635 29292 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.230388 29292 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:15.230403 29425 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:15.230676 29095 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.230814 30166 raft_consensus.cc:2781] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.230934 30166 raft_consensus.cc:686] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:15.231077 30166 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.232821 28875 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:01:15.233392 30178 raft_consensus.cc:482] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.233475 30178 raft_consensus.cc:504] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.233814 30178 leader_election.cc:288] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.234155 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4087c0e5443093e856fe676839b5" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:15.234192 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4087c0e5443093e856fe676839b5" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:15.234272 29292 raft_consensus.cc:2454] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:15.234329 29160 raft_consensus.cc:2454] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:15.234647 28962 leader_election.cc:302] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.234823 30178 raft_consensus.cc:2781] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.234879 30178 raft_consensus.cc:482] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.234920 30178 raft_consensus.cc:3037] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.235667 30178 raft_consensus.cc:504] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.235961 30178 leader_election.cc:288] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.236241 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4087c0e5443093e856fe676839b5" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:15.236316 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "252f4087c0e5443093e856fe676839b5" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:15.236356 29160 raft_consensus.cc:3037] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.236418 29292 raft_consensus.cc:3037] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.237008 29160 raft_consensus.cc:2454] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:15.237026 29292 raft_consensus.cc:2454] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:15.237263 28961 leader_election.cc:302] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:15.237425 30178 raft_consensus.cc:2781] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.237538 30178 raft_consensus.cc:686] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:15.237675 30178 consensus_queue.cc:237] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.239079 28875 catalog_manager.cc:5484] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:01:15.344579 30039 raft_consensus.cc:482] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.344698 30039 raft_consensus.cc:504] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.345074 30039 leader_election.cc:288] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:15.345453 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0669ab13586a48ff9382fe1b80f47553" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:15.345508 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0669ab13586a48ff9382fe1b80f47553" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:15.345577 29028 raft_consensus.cc:2454] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:15.345677 29425 raft_consensus.cc:2454] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:15.345891 29227 leader_election.cc:302] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.346083 30039 raft_consensus.cc:2781] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.346141 30039 raft_consensus.cc:482] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.346189 30039 raft_consensus.cc:3037] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.347004 30039 raft_consensus.cc:504] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.347332 30039 leader_election.cc:288] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:15.347605 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0669ab13586a48ff9382fe1b80f47553" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:15.347621 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0669ab13586a48ff9382fe1b80f47553" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:15.347693 29028 raft_consensus.cc:3037] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.347740 29425 raft_consensus.cc:3037] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.348567 29028 raft_consensus.cc:2454] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:15.348668 29425 raft_consensus.cc:2454] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:15.348820 29227 leader_election.cc:302] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.348971 30039 raft_consensus.cc:2781] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.349154 30039 raft_consensus.cc:686] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:15.349318 30039 consensus_queue.cc:237] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:15.350898 28875 catalog_manager.cc:5484] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:01:15.480774 30257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:15.481003 30257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:15.483800 30257 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:01:15.565644 29292 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.565702 29160 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.566057 30178 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.566179 30088 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.572752 29425 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.572957 29424 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.573330 30039 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.577210 30166 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.579394 29292 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:15.583835 30166 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:15.584658 29028 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:01:15.589385 30255 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.316 ms
time per row: 0.0556581 ms
I20240502 14:01:15.633445 28879 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:15.638409 29140 tablet_service.cc:1467] Processing CreateTablet for tablet f05a1a0bf1314d098f2bf65df58222e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.638485 29404 tablet_service.cc:1467] Processing CreateTablet for tablet f05a1a0bf1314d098f2bf65df58222e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.638710 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.638769 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.640364 30244 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:15.640440 30241 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:15.640617 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 400da91cbe234c79859b23ebe1643eb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.640870 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400da91cbe234c79859b23ebe1643eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.640851 29008 tablet_service.cc:1467] Processing CreateTablet for tablet f05a1a0bf1314d098f2bf65df58222e2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.641096 29008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.641206 30241 tablet_bootstrap.cc:654] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.641206 30244 tablet_bootstrap.cc:654] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.641939 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 5a91fd318f0844c1961f16823a7ccc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.642109 30241 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:15.642138 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.642171 30241 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.642593 30241 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.642688 30241 raft_consensus.cc:374] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.642717 30241 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.642823 30241 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.642946 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 5a91fd318f0844c1961f16823a7ccc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.643009 30242 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:15.643066 30241 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.643147 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.643785 30242 tablet_bootstrap.cc:654] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.643942 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 400da91cbe234c79859b23ebe1643eb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.644134 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400da91cbe234c79859b23ebe1643eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.644445 30240 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:15.645197 30240 tablet_bootstrap.cc:654] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.645380 30242 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:15.645442 30242 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:15.645862 30242 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.645963 30242 raft_consensus.cc:374] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.645995 30242 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.646117 30242 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.646369 30242 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.646538 30242 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:15.646554 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 400da91cbe234c79859b23ebe1643eb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.646782 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400da91cbe234c79859b23ebe1643eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.646993 30244 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:15.647068 30244 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:01:15.647373 30242 tablet_bootstrap.cc:654] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.647476 30244 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.647575 30244 raft_consensus.cc:374] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.647629 30244 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.647781 30244 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.648072 30244 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.648381 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 5a91fd318f0844c1961f16823a7ccc88 (DEFAULT_TABLE table=rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:15.648579 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:15.648885 30241 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:15.649667 30241 tablet_bootstrap.cc:654] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.650043 30242 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:15.650122 30242 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:15.650511 30242 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.650619 30242 raft_consensus.cc:374] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.650655 30242 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.650779 30241 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:15.650765 30242 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.650853 30241 ts_tablet_manager.cc:1367] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:15.650905 30244 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:15.651015 30242 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.651139 30242 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:15.651299 30241 raft_consensus.cc:348] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.651387 30241 raft_consensus.cc:374] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.651417 30241 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.651508 30241 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.651716 30244 tablet_bootstrap.cc:654] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.651743 30241 ts_tablet_manager.cc:1398] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.651950 30242 tablet_bootstrap.cc:654] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.652937 30244 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:15.653003 30244 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:15.653357 30240 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:15.653390 30244 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.653465 30240 ts_tablet_manager.cc:1367] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:01:15.653493 30244 raft_consensus.cc:374] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.653524 30244 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.653645 30244 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.653865 30244 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.653873 30240 raft_consensus.cc:348] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.653986 30240 raft_consensus.cc:374] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.654026 30240 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.654142 30240 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.654420 30240 ts_tablet_manager.cc:1398] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.654589 30240 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:15.654716 30242 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:15.654780 30242 ts_tablet_manager.cc:1367] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:15.655184 30242 raft_consensus.cc:348] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.655275 30242 raft_consensus.cc:374] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.655305 30242 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.655459 30240 tablet_bootstrap.cc:654] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:15.655421 30242 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.655725 30242 ts_tablet_manager.cc:1398] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:15.656821 30240 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:15.656896 30240 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:01:15.657227 30240 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.657313 30240 raft_consensus.cc:374] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:15.657346 30240 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:15.657470 30240 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.657727 30240 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:15.705770 30291 raft_consensus.cc:482] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.705917 30291 raft_consensus.cc:504] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.707099 30291 leader_election.cc:288] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:15.711016 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05a1a0bf1314d098f2bf65df58222e2" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:15.711180 29160 raft_consensus.cc:2454] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:15.711534 29358 leader_election.cc:302] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:15.711669 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05a1a0bf1314d098f2bf65df58222e2" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:15.711781 30291 raft_consensus.cc:2781] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.711799 29028 raft_consensus.cc:2454] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:15.711838 30291 raft_consensus.cc:482] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.711880 30291 raft_consensus.cc:3037] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.712998 30291 raft_consensus.cc:504] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.713382 30291 leader_election.cc:288] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:15.713728 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05a1a0bf1314d098f2bf65df58222e2" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:15.713737 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05a1a0bf1314d098f2bf65df58222e2" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:15.713827 29028 raft_consensus.cc:3037] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.713827 29160 raft_consensus.cc:3037] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.714742 29028 raft_consensus.cc:2454] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:15.714742 29160 raft_consensus.cc:2454] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:15.715065 29359 leader_election.cc:302] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:15.715255 30291 raft_consensus.cc:2781] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.715380 30291 raft_consensus.cc:686] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:15.715690 30291 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.717701 28884 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 0 to 1, leader changed from <none> to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:01:15.746376 30295 raft_consensus.cc:482] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.746500 30295 raft_consensus.cc:504] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.746865 30295 leader_election.cc:288] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.747151 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a91fd318f0844c1961f16823a7ccc88" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:15.747231 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a91fd318f0844c1961f16823a7ccc88" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:15.747287 29425 raft_consensus.cc:2454] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:15.747354 29293 raft_consensus.cc:2454] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:15.747635 28960 leader_election.cc:302] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:15.747800 30295 raft_consensus.cc:2781] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.747859 30295 raft_consensus.cc:482] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.747908 30295 raft_consensus.cc:3037] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.748842 30295 raft_consensus.cc:504] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.749164 30295 leader_election.cc:288] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:15.749423 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a91fd318f0844c1961f16823a7ccc88" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:15.749476 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a91fd318f0844c1961f16823a7ccc88" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:15.749534 29425 raft_consensus.cc:3037] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.749568 29293 raft_consensus.cc:3037] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.750406 29425 raft_consensus.cc:2454] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:15.750406 29293 raft_consensus.cc:2454] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:15.750631 28960 leader_election.cc:302] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:15.750777 30295 raft_consensus.cc:2781] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.750854 30295 raft_consensus.cc:686] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:15.751008 30295 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:15.752466 28884 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:01:15.902856 30294 raft_consensus.cc:482] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:15.902984 30294 raft_consensus.cc:504] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.903415 30294 leader_election.cc:288] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:15.903792 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:15.903828 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:15.903932 29160 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:15.903967 29028 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:15.904176 29226 leader_election.cc:302] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.904382 30294 raft_consensus.cc:2781] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:15.904440 30294 raft_consensus.cc:482] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:15.904498 30294 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.905408 30294 raft_consensus.cc:504] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.905731 30294 leader_election.cc:288] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:15.906071 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:15.906075 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:15.906175 29160 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.906183 29028 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:15.907028 29028 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:15.907074 29160 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:15.907306 29226 leader_election.cc:302] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:15.907558 30294 raft_consensus.cc:2781] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:15.907675 30294 raft_consensus.cc:686] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:15.907835 30294 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:15.909468 28884 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:01:16.041586 30314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:16.041815 30314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:16.044592 30314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:01:16.095321 30291 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.102790 30291 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.128404 29161 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.129194 29027 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.129273 30294 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.129758 29425 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.129776 30294 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.131825 30178 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.131983 29293 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.132478 30178 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 102.254 ms
time per row: 0.051127 ms
I20240502 14:01:16.190248 28884 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:16.195020 29140 tablet_service.cc:1467] Processing CreateTablet for tablet a75497cf22014c0cb9d079b448284b3d (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.195078 29273 tablet_service.cc:1467] Processing CreateTablet for tablet a75497cf22014c0cb9d079b448284b3d (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.195360 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.195375 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.195636 29405 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1230a7230f48558da1069a5988b981 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.195900 29405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.196401 29403 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda4193aaaf459fb5049ff49541229a (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.196619 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.197666 30374 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:16.197832 30376 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:16.198100 30375 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:16.198513 30374 tablet_bootstrap.cc:654] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.198571 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda4193aaaf459fb5049ff49541229a (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.198813 30376 tablet_bootstrap.cc:654] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.198819 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.199100 30375 tablet_bootstrap.cc:654] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.199952 30375 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:16.200006 30375 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:16.200358 30375 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.200452 30375 raft_consensus.cc:374] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.200452 30381 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:16.200479 30375 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.200549 29404 tablet_service.cc:1467] Processing CreateTablet for tablet a75497cf22014c0cb9d079b448284b3d (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.200585 30375 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.200728 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.201238 30381 tablet_bootstrap.cc:654] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.201333 30375 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.201781 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1230a7230f48558da1069a5988b981 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.201805 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 1dda4193aaaf459fb5049ff49541229a (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.202003 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.202118 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.202189 30381 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:16.202271 30381 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:16.202651 30381 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.202759 30381 raft_consensus.cc:374] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.202797 30381 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.202895 30381 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.203001 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1230a7230f48558da1069a5988b981 (DEFAULT_TABLE table=rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.203117 30381 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.203202 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.204833 30376 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:16.204936 30376 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:01:16.205394 30376 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.205493 30376 raft_consensus.cc:374] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.205526 30376 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.205600 30374 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:16.205633 30376 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.205689 30374 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:01:16.205819 30375 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:16.205915 30376 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.206074 30376 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:16.206135 30374 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.206238 30374 raft_consensus.cc:374] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.206272 30374 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.206372 30374 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.206612 30374 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.206789 30374 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:16.206823 30376 tablet_bootstrap.cc:654] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.207545 30374 tablet_bootstrap.cc:654] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.207731 30375 tablet_bootstrap.cc:654] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.208554 30374 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:16.208632 30374 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:16.208247 30376 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:16.209017 30376 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:16.209004 30374 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.209097 30374 raft_consensus.cc:374] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.209129 30374 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.209240 30375 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:16.209218 30374 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.209303 30375 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:01:16.209376 30376 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.209443 30374 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:16.209453 30376 raft_consensus.cc:374] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.209481 30376 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.209573 30374 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:16.209614 30376 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.209673 30375 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.209758 30375 raft_consensus.cc:374] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.209786 30375 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.209882 30375 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.210139 30375 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:16.210340 30374 tablet_bootstrap.cc:654] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.211480 30374 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:16.211552 30376 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:16.211562 30374 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:16.211699 30376 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:16.211880 30374 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.211963 30374 raft_consensus.cc:374] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.211994 30374 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.212086 30374 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.212347 30374 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:16.212404 30376 tablet_bootstrap.cc:654] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.213479 30376 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:16.213589 30376 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:01:16.213965 30376 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.214046 30376 raft_consensus.cc:374] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.214076 30376 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.214180 30376 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.214388 30376 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.320875 30294 raft_consensus.cc:482] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.321013 30294 raft_consensus.cc:504] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.321442 30294 leader_election.cc:288] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.321832 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75497cf22014c0cb9d079b448284b3d" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:16.321862 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75497cf22014c0cb9d079b448284b3d" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:16.321991 29425 raft_consensus.cc:2454] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.321990 29161 raft_consensus.cc:2454] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.322332 29226 leader_election.cc:302] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:16.322520 30294 raft_consensus.cc:2781] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.322575 30294 raft_consensus.cc:482] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.322604 30294 raft_consensus.cc:3037] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.323575 30294 raft_consensus.cc:504] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.323956 30294 leader_election.cc:288] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.324306 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75497cf22014c0cb9d079b448284b3d" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:16.324316 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a75497cf22014c0cb9d079b448284b3d" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:16.324422 29161 raft_consensus.cc:3037] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.324424 29425 raft_consensus.cc:3037] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.325412 29161 raft_consensus.cc:2454] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.325413 29425 raft_consensus.cc:2454] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.325783 29225 leader_election.cc:302] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:16.325999 30294 raft_consensus.cc:2781] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.326089 30294 raft_consensus.cc:686] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:16.326222 30294 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.327844 28884 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:16.372337 30166 raft_consensus.cc:482] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.372462 30166 raft_consensus.cc:504] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.372807 30166 leader_election.cc:288] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.373123 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:16.373253 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:16.373291 29028 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:16.373387 29425 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:16.373557 29095 leader_election.cc:302] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:16.373734 30166 raft_consensus.cc:2781] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.373806 30166 raft_consensus.cc:482] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.373831 30166 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.374663 30166 raft_consensus.cc:504] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.375017 30166 leader_election.cc:288] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.375319 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:16.375329 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:16.375432 29425 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.375432 29028 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.376368 29028 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:16.376368 29425 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:16.376636 29095 leader_election.cc:302] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:16.376821 30389 raft_consensus.cc:2781] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.376896 30389 raft_consensus.cc:686] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:16.377049 30389 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.378644 28884 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:16.456557 30365 raft_consensus.cc:482] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.456676 30365 raft_consensus.cc:504] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.457124 30365 leader_election.cc:288] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:16.457561 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1230a7230f48558da1069a5988b981" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:16.457731 29161 raft_consensus.cc:2454] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:16.458034 29358 leader_election.cc:302] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:16.458236 30365 raft_consensus.cc:2781] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.458302 30365 raft_consensus.cc:482] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.458333 30365 raft_consensus.cc:3037] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.459193 30365 raft_consensus.cc:504] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.459513 30365 leader_election.cc:288] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:16.459796 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1230a7230f48558da1069a5988b981" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:16.459911 29161 raft_consensus.cc:3037] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.460765 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1230a7230f48558da1069a5988b981" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:16.460845 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1230a7230f48558da1069a5988b981" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:16.460912 29293 raft_consensus.cc:2454] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:16.477023 29161 raft_consensus.cc:2454] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:16.477339 29358 leader_election.cc:302] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters: cd692a8958c241fca3aec5cca3a0ded5
I20240502 14:01:16.477523 30365 raft_consensus.cc:2781] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.477615 30365 raft_consensus.cc:686] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:16.477756 30365 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:16.479466 28884 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 0 to 1, leader changed from <none> to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:01:16.599895 30393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:16.600123 30393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:16.603107 30393 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:01:16.679809 29424 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.680282 30389 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.680629 29292 raft_consensus.cc:3037] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.682029 29028 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.682104 29161 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.682454 30389 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.683554 30365 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.689797 29425 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.691825 30294 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.697257 29292 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.697697 30291 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.702069 29161 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:16.702461 30255 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:16.714970 30429 mvcc.cc:205] Tried to move back new op lower bound from 7023241120516239360 to 7023241119033384960. Current Snapshot: MvccSnapshot[applied={T|T < 7023241120516239360}]
INSERT report
rows total: 2000
time total: 95.7773 ms
time per row: 0.0478886 ms
I20240502 14:01:16.746191 28879 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:16.751650 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 785828dee4594ca8813b9418cc5dba2d (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.751976 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785828dee4594ca8813b9418cc5dba2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.752801 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.752816 29271 tablet_service.cc:1467] Processing CreateTablet for tablet 20e960d4e55047918a1dad82fae21ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.753072 29271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e960d4e55047918a1dad82fae21ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.753072 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0176fb1a25a94c1c9d8cf0dc63577e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.753988 30456 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:16.754132 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.754350 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0176fb1a25a94c1c9d8cf0dc63577e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.755070 30456 tablet_bootstrap.cc:654] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.755375 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 785828dee4594ca8813b9418cc5dba2d (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.755551 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785828dee4594ca8813b9418cc5dba2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.756332 30456 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:16.756459 30456 ts_tablet_manager.cc:1367] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:16.756654 29007 tablet_service.cc:1467] Processing CreateTablet for tablet 20e960d4e55047918a1dad82fae21ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.756783 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 20e960d4e55047918a1dad82fae21ecd (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.756867 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e960d4e55047918a1dad82fae21ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.756783 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 785828dee4594ca8813b9418cc5dba2d (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.757033 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e960d4e55047918a1dad82fae21ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.757122 30456 raft_consensus.cc:348] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.757293 30456 raft_consensus.cc:374] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.757381 30456 raft_consensus.cc:729] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.757441 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785828dee4594ca8813b9418cc5dba2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.757607 30456 consensus_queue.cc:260] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.758013 30456 ts_tablet_manager.cc:1398] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.758599 30456 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:16.759462 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 (DEFAULT_TABLE table=rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:16.759629 30456 tablet_bootstrap.cc:654] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.759690 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0176fb1a25a94c1c9d8cf0dc63577e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:16.760526 30457 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:16.761190 30457 tablet_bootstrap.cc:654] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.761422 30460 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:16.761839 30458 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:16.762281 30460 tablet_bootstrap.cc:654] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.762533 30457 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:16.762645 30457 ts_tablet_manager.cc:1367] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:16.762657 30458 tablet_bootstrap.cc:654] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.763059 30457 raft_consensus.cc:348] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.763168 30457 raft_consensus.cc:374] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.763200 30457 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.763321 30460 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:16.763324 30457 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.763396 30460 ts_tablet_manager.cc:1367] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:16.763567 30457 ts_tablet_manager.cc:1398] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.763634 30458 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:16.763701 30458 ts_tablet_manager.cc:1367] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:16.763796 30460 raft_consensus.cc:348] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.763896 30456 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:16.763906 30460 raft_consensus.cc:374] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.763934 30460 raft_consensus.cc:729] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.763963 30456 ts_tablet_manager.cc:1367] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:16.764052 30460 consensus_queue.cc:260] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.764087 30458 raft_consensus.cc:348] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.764161 30458 raft_consensus.cc:374] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.764189 30458 raft_consensus.cc:729] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.764360 30460 ts_tablet_manager.cc:1398] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.764331 30458 consensus_queue.cc:260] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.764348 30456 raft_consensus.cc:348] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.764526 30456 raft_consensus.cc:374] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.764575 30456 raft_consensus.cc:729] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.764587 30458 ts_tablet_manager.cc:1398] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.764703 30456 consensus_queue.cc:260] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.764752 30458 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:16.765338 30456 ts_tablet_manager.cc:1398] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.765489 30456 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:16.765832 30458 tablet_bootstrap.cc:654] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.766209 30456 tablet_bootstrap.cc:654] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.767060 30460 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:16.767832 30460 tablet_bootstrap.cc:654] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.769245 30460 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:16.769327 30460 ts_tablet_manager.cc:1367] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:16.769728 30460 raft_consensus.cc:348] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.769826 30460 raft_consensus.cc:374] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.769881 30460 raft_consensus.cc:729] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.770013 30460 consensus_queue.cc:260] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.770252 30460 ts_tablet_manager.cc:1398] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.770699 30456 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:16.770785 30456 ts_tablet_manager.cc:1367] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:16.770906 30458 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:16.770982 30458 ts_tablet_manager.cc:1367] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:01:16.771238 30456 raft_consensus.cc:348] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.771348 30456 raft_consensus.cc:374] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.771392 30456 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.771378 30458 raft_consensus.cc:348] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.771510 30456 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.771782 30456 ts_tablet_manager.cc:1398] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.771916 30458 raft_consensus.cc:374] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.771968 30458 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.772078 30458 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.772459 30458 ts_tablet_manager.cc:1398] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:16.772639 30458 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:16.773722 30458 tablet_bootstrap.cc:654] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:16.774891 30458 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:16.774962 30458 ts_tablet_manager.cc:1367] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:16.775245 30458 raft_consensus.cc:348] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.775321 30458 raft_consensus.cc:374] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:16.775357 30458 raft_consensus.cc:729] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:16.775432 30458 consensus_queue.cc:260] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.775629 30458 ts_tablet_manager.cc:1398] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:16.798663 30294 raft_consensus.cc:482] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.798775 30294 raft_consensus.cc:504] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.799050 30294 leader_election.cc:288] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.799399 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:16.799413 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:16.799532 29028 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.799539 29424 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.799787 29225 leader_election.cc:302] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:16.799950 30294 raft_consensus.cc:2781] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.800005 30294 raft_consensus.cc:482] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.800029 30294 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.800920 30294 raft_consensus.cc:504] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.801218 30294 leader_election.cc:288] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:16.801470 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:16.801573 29028 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.802258 29028 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.802466 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:16.802502 29227 leader_election.cc:302] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:16.802577 29424 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.802650 30294 raft_consensus.cc:2781] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.802724 30294 raft_consensus.cc:686] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:16.802860 30294 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:16.803786 29424 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.804338 30294 raft_consensus.cc:482] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.804426 30294 raft_consensus.cc:504] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.804752 30294 leader_election.cc:288] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:16.804988 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e960d4e55047918a1dad82fae21ecd" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:16.805107 29028 raft_consensus.cc:2454] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.805042 28884 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:01:16.805220 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e960d4e55047918a1dad82fae21ecd" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:16.805357 29161 raft_consensus.cc:2454] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:16.805369 29227 leader_election.cc:302] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:16.805502 30294 raft_consensus.cc:2781] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.805546 30294 raft_consensus.cc:482] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.805579 30294 raft_consensus.cc:3037] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.806231 30294 raft_consensus.cc:504] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.806569 30294 leader_election.cc:288] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:16.806830 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e960d4e55047918a1dad82fae21ecd" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:16.806874 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20e960d4e55047918a1dad82fae21ecd" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:16.806932 29028 raft_consensus.cc:3037] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.807000 29161 raft_consensus.cc:3037] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.807735 29028 raft_consensus.cc:2454] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.807782 29161 raft_consensus.cc:2454] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:16.808017 29227 leader_election.cc:302] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:16.808207 30294 raft_consensus.cc:2781] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.808301 30294 raft_consensus.cc:686] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:16.808430 30294 consensus_queue.cc:237] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.809711 28884 catalog_manager.cc:5484] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:01:16.813810 30178 raft_consensus.cc:482] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:16.813896 30178 raft_consensus.cc:504] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.814210 30178 leader_election.cc:288] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:16.814507 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785828dee4594ca8813b9418cc5dba2d" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:16.814543 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785828dee4594ca8813b9418cc5dba2d" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:16.814666 29292 raft_consensus.cc:2454] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:16.814679 29161 raft_consensus.cc:2454] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:16.814911 28962 leader_election.cc:302] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:16.815066 30178 raft_consensus.cc:2781] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:16.815120 30178 raft_consensus.cc:482] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:16.815163 30178 raft_consensus.cc:3037] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.815899 30178 raft_consensus.cc:504] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.816197 30178 leader_election.cc:288] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:16.816511 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785828dee4594ca8813b9418cc5dba2d" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:16.816500 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "785828dee4594ca8813b9418cc5dba2d" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:16.816601 29161 raft_consensus.cc:3037] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.816612 29292 raft_consensus.cc:3037] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:16.817467 29292 raft_consensus.cc:2454] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:16.817467 29161 raft_consensus.cc:2454] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:16.817744 28961 leader_election.cc:302] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:16.817916 30178 raft_consensus.cc:2781] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:16.818107 30178 raft_consensus.cc:686] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:16.818250 30178 consensus_queue.cc:237] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:16.819803 28876 catalog_manager.cc:5484] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:01:16.939754 30474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:16.939972 30474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:16.942703 30474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 14:01:17.044354 29028 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.044816 30294 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.045953 29424 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.046324 29161 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.047873 29027 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.047909 29292 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.048249 29161 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.048704 30255 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.048935 30294 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.049260 30178 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.053208 30294 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.054697 30503 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.069816 30435 mvcc.cc:205] Tried to move back new op lower bound from 7023241121974689792 to 7023241121008472064. Current Snapshot: MvccSnapshot[applied={T|T < 7023241121974689792}]
INSERT report
rows total: 2000
time total: 114.17 ms
time per row: 0.0570848 ms
I20240502 14:01:17.106310 28878 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:17.111860 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 81b2d5d0f4044106bed079c3faa72447 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.112200 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.112233 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 81b2d5d0f4044106bed079c3faa72447 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.112550 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.114174 30456 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:17.114264 30457 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:17.114792 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 81b2d5d0f4044106bed079c3faa72447 (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.115060 30456 tablet_bootstrap.cc:654] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.115056 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.115216 30457 tablet_bootstrap.cc:654] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.116672 30457 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:17.116751 30457 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:17.116864 30460 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:17.117025 29141 tablet_service.cc:1467] Processing CreateTablet for tablet a476d1664de7456ab21ddbf2e12b4f0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.117209 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a476d1664de7456ab21ddbf2e12b4f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.117220 30457 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.117334 30457 raft_consensus.cc:374] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.117362 30457 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.117470 30457 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.117692 30460 tablet_bootstrap.cc:654] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.117738 30457 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:17.118865 30460 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:17.118945 30460 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:01:17.119349 30460 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.119447 30460 raft_consensus.cc:374] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.119488 30460 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.119554 29404 tablet_service.cc:1467] Processing CreateTablet for tablet dcc536794ffb453495d025f29ea152bc (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.119586 30460 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.119757 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc536794ffb453495d025f29ea152bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.119822 30460 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:01:17.119941 30460 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:17.120551 29403 tablet_service.cc:1467] Processing CreateTablet for tablet a476d1664de7456ab21ddbf2e12b4f0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.120699 30460 tablet_bootstrap.cc:654] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.120746 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a476d1664de7456ab21ddbf2e12b4f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.121888 30460 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:17.121963 30460 ts_tablet_manager.cc:1367] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:17.122427 30460 raft_consensus.cc:348] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.122561 30460 raft_consensus.cc:374] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.122602 30460 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.122603 30457 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:17.122689 30460 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.123032 30460 ts_tablet_manager.cc:1398] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:17.123453 30457 tablet_bootstrap.cc:654] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.124126 29273 tablet_service.cc:1467] Processing CreateTablet for tablet dcc536794ffb453495d025f29ea152bc (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.124460 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc536794ffb453495d025f29ea152bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.124632 29272 tablet_service.cc:1467] Processing CreateTablet for tablet a476d1664de7456ab21ddbf2e12b4f0d (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.124859 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a476d1664de7456ab21ddbf2e12b4f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.126205 29006 tablet_service.cc:1467] Processing CreateTablet for tablet dcc536794ffb453495d025f29ea152bc (DEFAULT_TABLE table=rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.126457 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc536794ffb453495d025f29ea152bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.126508 30458 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:17.127326 30458 tablet_bootstrap.cc:654] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.127440 30457 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:17.127519 30457 ts_tablet_manager.cc:1367] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:01:17.128072 30457 raft_consensus.cc:348] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.128178 30457 raft_consensus.cc:374] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.128212 30457 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.128374 30457 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.128683 30457 ts_tablet_manager.cc:1398] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:17.128737 30458 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:17.128810 30458 ts_tablet_manager.cc:1367] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:17.128866 30457 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:17.129261 30458 raft_consensus.cc:348] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.129360 30458 raft_consensus.cc:374] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.129395 30458 raft_consensus.cc:729] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.129529 30458 consensus_queue.cc:260] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.129851 30458 ts_tablet_manager.cc:1398] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.130038 30458 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:17.130043 30456 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:17.130105 30456 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.000s
I20240502 14:01:17.130406 30456 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.130470 30456 raft_consensus.cc:374] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.130488 30456 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.130592 30456 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.130837 30456 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.130903 30458 tablet_bootstrap.cc:654] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.131000 30456 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:17.131059 30457 tablet_bootstrap.cc:654] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.131727 30456 tablet_bootstrap.cc:654] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.132086 30457 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:17.132167 30457 ts_tablet_manager.cc:1367] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:01:17.132680 30457 raft_consensus.cc:348] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.132779 30457 raft_consensus.cc:374] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.132813 30457 raft_consensus.cc:729] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.132937 30458 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:17.132926 30457 consensus_queue.cc:260] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.133003 30458 ts_tablet_manager.cc:1367] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:17.133201 30457 ts_tablet_manager.cc:1398] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.133423 30458 raft_consensus.cc:348] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.133539 30458 raft_consensus.cc:374] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.133571 30458 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.133704 30458 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.134047 30458 ts_tablet_manager.cc:1398] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.134411 30456 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:17.134485 30456 ts_tablet_manager.cc:1367] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:17.134995 30456 raft_consensus.cc:348] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.135078 30456 raft_consensus.cc:374] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.135110 30456 raft_consensus.cc:729] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.135255 30456 consensus_queue.cc:260] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.135574 30456 ts_tablet_manager.cc:1398] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.158615 30178 raft_consensus.cc:482] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.158715 30178 raft_consensus.cc:504] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.159101 30178 leader_election.cc:288] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.159451 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc536794ffb453495d025f29ea152bc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:17.159513 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc536794ffb453495d025f29ea152bc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:17.159587 29424 raft_consensus.cc:2454] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.159655 29292 raft_consensus.cc:2454] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.159830 28960 leader_election.cc:302] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.160001 30178 raft_consensus.cc:2781] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.160058 30178 raft_consensus.cc:482] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.160084 30178 raft_consensus.cc:3037] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.160897 30178 raft_consensus.cc:504] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.161180 30178 leader_election.cc:288] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.161445 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc536794ffb453495d025f29ea152bc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:17.161450 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc536794ffb453495d025f29ea152bc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:17.161533 29292 raft_consensus.cc:3037] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.161548 29424 raft_consensus.cc:3037] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.162209 29292 raft_consensus.cc:2454] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.162297 29424 raft_consensus.cc:2454] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.162453 28962 leader_election.cc:302] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:17.162606 30178 raft_consensus.cc:2781] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.162674 30178 raft_consensus.cc:686] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:17.162782 30178 consensus_queue.cc:237] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.164290 28879 catalog_manager.cc:5484] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:01:17.228605 30178 raft_consensus.cc:482] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.228757 30178 raft_consensus.cc:504] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.229178 30178 leader_election.cc:288] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.229575 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b2d5d0f4044106bed079c3faa72447" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:17.229585 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b2d5d0f4044106bed079c3faa72447" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:17.229739 29160 raft_consensus.cc:2454] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.229749 29424 raft_consensus.cc:2454] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.230067 28960 leader_election.cc:302] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.230248 30178 raft_consensus.cc:2781] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.230304 30178 raft_consensus.cc:482] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.230334 30178 raft_consensus.cc:3037] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.231159 30178 raft_consensus.cc:504] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.231484 30178 leader_election.cc:288] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.231817 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b2d5d0f4044106bed079c3faa72447" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:17.231803 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b2d5d0f4044106bed079c3faa72447" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:17.231920 29424 raft_consensus.cc:3037] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.232045 29160 raft_consensus.cc:3037] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.232762 29424 raft_consensus.cc:2454] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.232925 29160 raft_consensus.cc:2454] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.233045 28960 leader_election.cc:302] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.233214 30178 raft_consensus.cc:2781] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.233299 30178 raft_consensus.cc:686] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:17.233443 30178 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.235167 28879 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:01:17.417878 30441 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.418011 30441 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.418365 30441 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.418735 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:17.418745 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:17.418854 29292 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:17.418866 29160 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:17.419126 29359 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.419277 30441 raft_consensus.cc:2781] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.419328 30441 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.419351 30441 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.420087 30441 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.420382 30441 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.420684 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:17.420689 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:17.420778 29160 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.420783 29292 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.421470 29160 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:17.421484 29292 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:17.421720 29358 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.421885 30441 raft_consensus.cc:2781] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.421998 30441 raft_consensus.cc:686] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:17.422134 30441 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.424018 28879 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 0 to 1, leader changed from <none> to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:01:17.546000 30178 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.549975 30503 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:01:17.600708 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:17.600922 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:17.603950 30526 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 14:01:17.686887 29425 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.687510 29160 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.688030 30472 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.688485 30472 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.688843 29293 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.689244 30291 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.690042 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:17.691586 30441 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:17.698724 30555 mvcc.cc:205] Tried to move back new op lower bound from 7023241124595544064 to 7023241122749501440. Current Snapshot: MvccSnapshot[applied={T|T < 7023241124595544064 or (T in {7023241124595544064})}]
I20240502 14:01:17.704401 30431 mvcc.cc:205] Tried to move back new op lower bound from 7023241124595544064 to 7023241122749501440. Current Snapshot: MvccSnapshot[applied={T|T < 7023241124595544064}]
INSERT report
rows total: 2000
time total: 100.297 ms
time per row: 0.0501484 ms
I20240502 14:01:17.755256 28884 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:17.760460 29404 tablet_service.cc:1467] Processing CreateTablet for tablet ed713f89a4fd47d79f03711dd14c1334 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.760540 29006 tablet_service.cc:1467] Processing CreateTablet for tablet ed713f89a4fd47d79f03711dd14c1334 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.760772 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed713f89a4fd47d79f03711dd14c1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.760792 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed713f89a4fd47d79f03711dd14c1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.760799 29141 tablet_service.cc:1467] Processing CreateTablet for tablet ed713f89a4fd47d79f03711dd14c1334 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.761098 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed713f89a4fd47d79f03711dd14c1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.762997 30582 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:17.763368 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 19784717201a464c84607d2a43e651c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.763625 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19784717201a464c84607d2a43e651c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.763900 30582 tablet_bootstrap.cc:654] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.764271 29006 tablet_service.cc:1467] Processing CreateTablet for tablet 19784717201a464c84607d2a43e651c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.764451 30581 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:17.764545 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19784717201a464c84607d2a43e651c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.764513 29273 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffcf19e66e4adca28d87cbe2245964 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.764766 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ffcf19e66e4adca28d87cbe2245964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.765156 29404 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffcf19e66e4adca28d87cbe2245964 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.765342 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ffcf19e66e4adca28d87cbe2245964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.765799 30580 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:17.766582 29007 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffcf19e66e4adca28d87cbe2245964 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.766644 30580 tablet_bootstrap.cc:654] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.766794 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ffcf19e66e4adca28d87cbe2245964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.766764 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 19784717201a464c84607d2a43e651c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:17.766976 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19784717201a464c84607d2a43e651c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:17.767589 30580 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:17.767668 30580 ts_tablet_manager.cc:1367] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:17.768047 30581 tablet_bootstrap.cc:654] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.768180 30580 raft_consensus.cc:348] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.768307 30580 raft_consensus.cc:374] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.768344 30580 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.768496 30580 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.768841 30580 ts_tablet_manager.cc:1398] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.769024 30582 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:17.769032 30580 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:17.769095 30582 ts_tablet_manager.cc:1367] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:01:17.769445 30582 raft_consensus.cc:348] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.769536 30582 raft_consensus.cc:374] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.769578 30582 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.769673 30582 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.769925 30580 tablet_bootstrap.cc:654] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.769939 30582 ts_tablet_manager.cc:1398] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.770090 30582 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:17.770146 30584 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:17.770992 30584 tablet_bootstrap.cc:654] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.771423 30580 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:17.771502 30580 ts_tablet_manager.cc:1367] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:17.771776 30582 tablet_bootstrap.cc:654] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.771944 30580 raft_consensus.cc:348] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.772063 30580 raft_consensus.cc:374] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.772099 30580 raft_consensus.cc:729] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.772234 30580 consensus_queue.cc:260] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.772764 30580 ts_tablet_manager.cc:1398] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.772881 30584 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:17.772977 30584 ts_tablet_manager.cc:1367] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:17.773052 30581 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:17.773123 30581 ts_tablet_manager.cc:1367] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:01:17.773303 30584 raft_consensus.cc:348] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.773416 30584 raft_consensus.cc:374] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.773516 30584 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.773552 30581 raft_consensus.cc:348] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.773713 30581 raft_consensus.cc:374] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.773748 30582 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:17.773701 30584 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.773849 30582 ts_tablet_manager.cc:1367] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:17.773993 30584 ts_tablet_manager.cc:1398] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.774243 30584 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:17.774232 30582 raft_consensus.cc:348] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.773787 30581 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.774372 30582 raft_consensus.cc:374] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.774444 30582 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.774562 30581 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.774585 30582 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.775096 30584 tablet_bootstrap.cc:654] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.775343 30582 ts_tablet_manager.cc:1398] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.776049 30581 ts_tablet_manager.cc:1398] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:17.776238 30581 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:17.777208 30581 tablet_bootstrap.cc:654] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.778201 30584 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:17.778326 30584 ts_tablet_manager.cc:1367] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:17.778803 30584 raft_consensus.cc:348] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.778895 30584 raft_consensus.cc:374] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.778925 30584 raft_consensus.cc:729] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.779035 30584 consensus_queue.cc:260] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.779350 30584 ts_tablet_manager.cc:1398] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:17.779896 30581 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:17.779973 30581 ts_tablet_manager.cc:1367] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:01:17.780542 30581 raft_consensus.cc:348] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.780684 30581 raft_consensus.cc:374] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.780728 30581 raft_consensus.cc:729] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.780864 30581 consensus_queue.cc:260] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.781116 30581 ts_tablet_manager.cc:1398] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.781311 30581 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:17.782328 30581 tablet_bootstrap.cc:654] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:17.783430 30581 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:17.783506 30581 ts_tablet_manager.cc:1367] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:17.783888 30581 raft_consensus.cc:348] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.783993 30581 raft_consensus.cc:374] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:17.784044 30581 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:17.784161 30581 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.784433 30581 ts_tablet_manager.cc:1398] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:17.930362 30503 raft_consensus.cc:482] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.930482 30503 raft_consensus.cc:504] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.930856 30503 leader_election.cc:288] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.931260 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19784717201a464c84607d2a43e651c2" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:17.931272 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19784717201a464c84607d2a43e651c2" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:17.931408 29161 raft_consensus.cc:2454] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.931421 29291 raft_consensus.cc:2454] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.931732 28961 leader_election.cc:302] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:17.931892 30503 raft_consensus.cc:2781] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.931947 30503 raft_consensus.cc:482] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.931989 30503 raft_consensus.cc:3037] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.932852 30503 raft_consensus.cc:504] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.933136 30503 leader_election.cc:288] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.933422 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19784717201a464c84607d2a43e651c2" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:17.933442 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19784717201a464c84607d2a43e651c2" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:17.933527 29161 raft_consensus.cc:3037] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.933527 29291 raft_consensus.cc:3037] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.934259 29161 raft_consensus.cc:2454] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.934319 29291 raft_consensus.cc:2454] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.934551 28961 leader_election.cc:302] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:17.934731 30503 raft_consensus.cc:2781] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.934814 30503 raft_consensus.cc:686] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:17.934973 30503 consensus_queue.cc:237] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:01:17.936755 28879 catalog_manager.cc:5484] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:01:17.949744 30571 raft_consensus.cc:482] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.949841 30571 raft_consensus.cc:504] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.950189 30571 leader_election.cc:288] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.950493 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:17.950531 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:17.950611 29027 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:17.950635 29161 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 0.
I20240502 14:01:17.950881 29358 leader_election.cc:302] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.951035 30571 raft_consensus.cc:2781] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.951104 30571 raft_consensus.cc:482] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.951138 30571 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.951828 30571 raft_consensus.cc:504] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.952097 30571 leader_election.cc:288] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Requested vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:17.952368 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:17.952417 29161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:17.952461 29027 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.952527 29161 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.953125 29027 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:17.953199 29161 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 1.
I20240502 14:01:17.953388 29359 leader_election.cc:302] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.953545 30571 raft_consensus.cc:2781] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.953655 30571 raft_consensus.cc:686] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:17.953790 30571 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:01:17.955541 28883 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 0 to 1, leader changed from <none> to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:01:17.969457 30503 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:17.969558 30503 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.969906 30503 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.970227 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:17.970264 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:17.970341 29424 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.970383 29291 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:17.970640 28962 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:17.970790 30503 raft_consensus.cc:2781] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:17.970844 30503 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:17.970882 30503 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.971681 30503 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.971989 30503 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:17.972316 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:17.972347 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:17.972412 29424 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.972450 29291 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:17.973151 29424 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.973207 29291 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:17.973409 28960 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:17.973546 30503 raft_consensus.cc:2781] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:17.973626 30503 raft_consensus.cc:686] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:17.973773 30503 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:17.975405 28883 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
W20240502 14:01:18.107791 30596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:18.108009 30596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:18.110911 30596 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 14:01:18.199196 29291 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.199394 29027 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.199409 29161 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.199743 29159 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.199946 30472 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.200126 29424 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.200188 30503 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.200948 29291 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.201627 30571 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.202024 30178 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.202405 30571 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.207381 30472 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 116.812 ms
time per row: 0.0584058 ms
I20240502 14:01:18.267516 28879 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56624:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:01:18.272629 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1adebdbd104796b927963e86a15295 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.272917 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1adebdbd104796b927963e86a15295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.273001 29272 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1adebdbd104796b927963e86a15295 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.273226 29272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1adebdbd104796b927963e86a15295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.273645 29140 tablet_service.cc:1467] Processing CreateTablet for tablet 15513e220e144883b216c4f3ac8ade0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.273841 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.275019 30582 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:18.275310 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 15513e220e144883b216c4f3ac8ade0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.275516 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.275588 29138 tablet_service.cc:1467] Processing CreateTablet for tablet c88f85f143d845a1b7066244c606f8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.275813 29138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.275853 30582 tablet_bootstrap.cc:654] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.275866 29404 tablet_service.cc:1467] Processing CreateTablet for tablet 15513e220e144883b216c4f3ac8ade0b (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.276067 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.276508 29403 tablet_service.cc:1467] Processing CreateTablet for tablet c88f85f143d845a1b7066244c606f8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.276700 29403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.277328 30580 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:18.277843 29007 tablet_service.cc:1467] Processing CreateTablet for tablet c88f85f143d845a1b7066244c606f8ba (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.278079 30580 tablet_bootstrap.cc:654] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.278080 29007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.278646 29141 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1adebdbd104796b927963e86a15295 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:01:18.278872 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1adebdbd104796b927963e86a15295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:18.278966 30584 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:18.279081 30582 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:18.279150 30582 ts_tablet_manager.cc:1367] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:01:18.279523 30582 raft_consensus.cc:348] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.279613 30582 raft_consensus.cc:374] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.279637 30582 raft_consensus.cc:729] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.279791 30584 tablet_bootstrap.cc:654] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.279760 30582 consensus_queue.cc:260] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.279994 30582 ts_tablet_manager.cc:1398] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.280129 30582 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:18.280143 30581 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:01:18.280877 30581 tablet_bootstrap.cc:654] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.281289 30582 tablet_bootstrap.cc:654] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.282433 30584 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:18.282514 30584 ts_tablet_manager.cc:1367] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:18.282552 30582 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:18.282615 30582 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:18.283000 30582 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.282982 30584 raft_consensus.cc:348] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.283120 30582 raft_consensus.cc:374] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.283121 30584 raft_consensus.cc:374] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.283172 30582 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.283200 30584 raft_consensus.cc:729] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.283267 30582 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.283327 30584 consensus_queue.cc:260] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.283552 30582 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.283630 30584 ts_tablet_manager.cc:1398] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.283727 30582 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:01:18.283787 30584 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:01:18.283891 30581 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: No bootstrap required, opened a new log
I20240502 14:01:18.283967 30581 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:01:18.284072 30580 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:18.284145 30580 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:01:18.284521 30582 tablet_bootstrap.cc:654] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.284540 30581 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.284631 30581 raft_consensus.cc:374] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.284659 30581 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.284618 30580 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.284726 30580 raft_consensus.cc:374] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.284755 30580 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.284761 30581 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.284879 30580 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.285039 30581 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.285182 30580 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:18.285341 30580 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:18.286036 30580 tablet_bootstrap.cc:654] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.286712 30584 tablet_bootstrap.cc:654] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.288471 30582 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: No bootstrap required, opened a new log
I20240502 14:01:18.288542 30580 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:18.288555 30582 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:18.288604 30580 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:01:18.288859 30584 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: No bootstrap required, opened a new log
I20240502 14:01:18.288930 30584 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:01:18.288924 30580 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.288913 30582 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.289013 30580 raft_consensus.cc:374] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.289013 30582 raft_consensus.cc:374] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.289048 30580 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.289048 30582 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.289140 30582 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.289171 30580 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.289336 30584 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.289381 30582 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.289423 30584 raft_consensus.cc:374] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.289461 30584 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.289533 30580 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.289577 30584 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.289796 30584 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:18.290203 30580 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:01:18.291180 30580 tablet_bootstrap.cc:654] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Neither blocks nor log segments found. Creating new log.
I20240502 14:01:18.293450 30580 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: No bootstrap required, opened a new log
I20240502 14:01:18.293511 30580 ts_tablet_manager.cc:1367] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:18.293848 30580 raft_consensus.cc:348] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.293928 30580 raft_consensus.cc:374] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:01:18.293958 30580 raft_consensus.cc:729] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:01:18.294075 30580 consensus_queue.cc:260] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.294277 30580 ts_tablet_manager.cc:1398] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:18.328838 30166 raft_consensus.cc:482] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:18.328979 30166 raft_consensus.cc:504] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.329435 30166 leader_election.cc:288] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:18.329787 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:18.329811 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:18.329932 29424 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:18.329957 29291 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 0.
I20240502 14:01:18.330183 29093 leader_election.cc:302] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:18.330327 30166 raft_consensus.cc:2781] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:18.330377 30166 raft_consensus.cc:482] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:18.330418 30166 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.331279 30166 raft_consensus.cc:504] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.331547 30166 leader_election.cc:288] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:18.331808 29291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:18.331820 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:18.331904 29424 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.331916 29291 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.332814 29424 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:18.332921 29291 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 1.
I20240502 14:01:18.333096 29093 leader_election.cc:302] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:18.333271 30166 raft_consensus.cc:2781] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:18.333346 30166 raft_consensus.cc:686] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:18.333474 30166 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.335114 28880 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:01:18.459475 30255 raft_consensus.cc:482] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:18.459611 30255 raft_consensus.cc:504] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.459968 30255 leader_election.cc:288] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:18.460378 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1adebdbd104796b927963e86a15295" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:18.460378 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1adebdbd104796b927963e86a15295" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:18.460522 29159 raft_consensus.cc:2454] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:18.460522 29424 raft_consensus.cc:2454] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 0.
I20240502 14:01:18.460793 29225 leader_election.cc:302] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:18.460980 30255 raft_consensus.cc:2781] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:18.461045 30255 raft_consensus.cc:482] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:18.461095 30255 raft_consensus.cc:3037] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.462031 30255 raft_consensus.cc:504] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.462270 30255 leader_election.cc:288] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:18.462545 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1adebdbd104796b927963e86a15295" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:18.462579 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1adebdbd104796b927963e86a15295" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:18.462684 29159 raft_consensus.cc:3037] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.462682 29424 raft_consensus.cc:3037] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.463600 29159 raft_consensus.cc:2454] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:18.463600 29424 raft_consensus.cc:2454] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 1.
I20240502 14:01:18.463850 29225 leader_election.cc:302] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:18.464008 30255 raft_consensus.cc:2781] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:18.464174 30255 raft_consensus.cc:686] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:18.464331 30255 consensus_queue.cc:237] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:01:18.466094 28880 catalog_manager.cc:5484] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:01:18.594273 30472 raft_consensus.cc:482] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:01:18.594414 30472 raft_consensus.cc:504] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.594841 30472 leader_election.cc:288] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:18.595214 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88f85f143d845a1b7066244c606f8ba" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:18.595233 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88f85f143d845a1b7066244c606f8ba" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:18.595355 29424 raft_consensus.cc:2454] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:18.595364 29159 raft_consensus.cc:2454] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 0.
I20240502 14:01:18.595669 28960 leader_election.cc:302] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:18.595850 30472 raft_consensus.cc:2781] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:01:18.595914 30472 raft_consensus.cc:482] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:01:18.595940 30472 raft_consensus.cc:3037] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.596767 30472 raft_consensus.cc:504] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.597086 30472 leader_election.cc:288] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:18.597370 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88f85f143d845a1b7066244c606f8ba" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:18.597440 29424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c88f85f143d845a1b7066244c606f8ba" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:18.597491 29159 raft_consensus.cc:3037] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.597589 29424 raft_consensus.cc:3037] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:01:18.598582 29159 raft_consensus.cc:2454] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:18.598585 29424 raft_consensus.cc:2454] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 1.
I20240502 14:01:18.598909 28961 leader_election.cc:302] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:18.599102 30178 raft_consensus.cc:2781] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:01:18.599211 30178 raft_consensus.cc:686] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:18.599402 30178 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:01:18.602497 28880 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:01:18.745884 30646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:18.746191 30646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:18.749131 30646 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 14:01:18.781945 30166 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.785807 30166 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.825243 29424 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.825707 30178 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.834061 29159 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.834414 30178 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.840014 29161 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.840204 29425 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:01:18.840389 30643 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:01:18.840585 30255 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 97.9697 ms
time per row: 0.0489848 ms
I20240502 14:01:18.884398 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.5:45769
--local_ip_for_outbound_sockets=127.16.217.5
--tserver_master_addrs=127.16.217.62:41117
--webserver_port=39135
--webserver_interface=127.16.217.5
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 14:01:18.986805 30700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:18.987002 30700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:18.987046 30700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:18.989981 30700 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:18.990041 30700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:18.990103 30700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.5
I20240502 14:01:18.993475 30700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.5:45769
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.217.5
--webserver_port=39135
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.30700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.5
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:18.994203 30700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:18.995271 30700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:19.003774 30706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:19.004062 30700 server_base.cc:1034] running on GCE node
W20240502 14:01:19.004190 30708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.004359 30705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:19.004782 30700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:19.005388 30700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:19.006582 30700 hybrid_clock.cc:648] HybridClock initialized: now 1714658479006554 us; error 41 us; skew 500 ppm
I20240502 14:01:19.013233 30700 webserver.cc:460] Webserver started at http://127.16.217.5:39135/ using document root <none> and password file <none>
I20240502 14:01:19.013860 30700 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:19.013945 30700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:19.017354 30700 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:01:19.019064 30715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.020030 30700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:19.020121 30700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "22dbbf207ba0440392e71a10c2ca78ca"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:19.020550 30700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:19.036010 30700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:19.036559 30700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:19.036697 30700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:19.037174 30700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:19.038012 30700 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:19.038055 30700 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.038105 30700 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:19.038130 30700 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.046553 30700 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.5:45769
I20240502 14:01:19.046622 30829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.5:45769 every 8 connection(s)
I20240502 14:01:19.047560 30700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:01:19.050447 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 30700
I20240502 14:01:19.050928 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.6:38737
--local_ip_for_outbound_sockets=127.16.217.6
--tserver_master_addrs=127.16.217.62:41117
--webserver_port=34871
--webserver_interface=127.16.217.6
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:19.056955 30830 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:19.057160 30830 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:19.057569 30830 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:19.058641 28879 ts_manager.cc:194] Re-registered known tserver with Master: 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:19.059183 28879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.5:52199
W20240502 14:01:19.151787 30833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.151960 30833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:19.151996 30833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.154611 30833 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:19.154678 30833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:19.154763 30833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.6
I20240502 14:01:19.158205 30833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.6:38737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.217.6
--webserver_port=34871
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.30833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.6
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:19.159013 30833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:19.160102 30833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:19.167588 30841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.167681 30839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.167984 30838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:19.168143 30833 server_base.cc:1034] running on GCE node
I20240502 14:01:19.168690 30833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:19.169243 30833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:19.170382 30833 hybrid_clock.cc:648] HybridClock initialized: now 1714658479170367 us; error 31 us; skew 500 ppm
I20240502 14:01:19.172067 30833 webserver.cc:460] Webserver started at http://127.16.217.6:34871/ using document root <none> and password file <none>
I20240502 14:01:19.172629 30833 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:19.172704 30833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:19.175770 30833 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:01:19.177536 30847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.178406 30833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:19.178503 30833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "12348ca540074b32912fdd6333484e0b"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:19.178879 30833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:19.197150 30833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:19.197662 30833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:19.197819 30833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:19.198323 30833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:19.199080 30833 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:19.199120 30833 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.199153 30833 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:19.199172 30833 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.206990 30833 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.6:38737
I20240502 14:01:19.207023 30962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.6:38737 every 8 connection(s)
I20240502 14:01:19.207906 30833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:01:19.215665 30963 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:19.215864 30963 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:19.216266 30963 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:19.216812 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 30833
I20240502 14:01:19.217182 28879 ts_manager.cc:194] Re-registered known tserver with Master: 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:01:19.217200 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.7:40969
--local_ip_for_outbound_sockets=127.16.217.7
--tserver_master_addrs=127.16.217.62:41117
--webserver_port=44249
--webserver_interface=127.16.217.7
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:19.217648 28879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.6:54287
W20240502 14:01:19.322031 30966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.322252 30966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:19.322311 30966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.324985 30966 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:19.325050 30966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:19.325140 30966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.7
I20240502 14:01:19.328344 30966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.7:40969
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.217.7
--webserver_port=44249
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.30966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.7
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:19.329159 30966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:19.330153 30966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:19.337203 30971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.337239 30972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.337347 30974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:19.337692 30966 server_base.cc:1034] running on GCE node
I20240502 14:01:19.338002 30966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:19.338452 30966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:19.339592 30966 hybrid_clock.cc:648] HybridClock initialized: now 1714658479339584 us; error 23 us; skew 500 ppm
I20240502 14:01:19.341288 30966 webserver.cc:460] Webserver started at http://127.16.217.7:44249/ using document root <none> and password file <none>
I20240502 14:01:19.341773 30966 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:19.341841 30966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:19.344806 30966 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:01:19.346751 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.347642 30966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:01:19.347765 30966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "33ebf308f6f24e6e8768ef5f14ddc785"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:19.348160 30966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:19.386803 30966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:19.387367 30966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:19.387528 30966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:19.388177 30966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:19.389037 30966 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:19.389093 30966 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.389130 30966 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:19.389148 30966 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.396740 30966 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.7:40969
I20240502 14:01:19.396791 31095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.7:40969 every 8 connection(s)
I20240502 14:01:19.397723 30966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:01:19.402645 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 30966
I20240502 14:01:19.402998 17252 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhNHehp/build/debug/bin/kudu
/tmp/dist-test-taskhNHehp/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.217.8:38573
--local_ip_for_outbound_sockets=127.16.217.8
--tserver_master_addrs=127.16.217.62:41117
--webserver_port=40059
--webserver_interface=127.16.217.8
--builtin_ntp_servers=127.16.217.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 14:01:19.405979 31096 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:19.406193 31096 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:19.406602 31096 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:19.407591 28879 ts_manager.cc:194] Re-registered known tserver with Master: 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:19.408551 28879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.7:55915
W20240502 14:01:19.503212 31099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.503401 31099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:01:19.503437 31099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.506455 31099 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 14:01:19.506528 31099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:01:19.506634 31099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.217.8
I20240502 14:01:19.510644 31099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.217.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.217.8:38573
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.217.8
--webserver_port=40059
--tserver_master_addrs=127.16.217.62:41117
--never_fsync=true
--heap_profile_path=/tmp/kudu.31099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.217.8
--log_dir=/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type DEBUG
built by None at 02 May 2024 13:43:11 UTC on 1a15035a6830
build id 89
I20240502 14:01:19.511426 31099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:01:19.512511 31099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:01:19.520259 31105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.520360 31106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:01:19.520423 31108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:01:19.520779 31099 server_base.cc:1034] running on GCE node
I20240502 14:01:19.521129 31099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:01:19.521647 31099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:01:19.522795 31099 hybrid_clock.cc:648] HybridClock initialized: now 1714658479522766 us; error 38 us; skew 500 ppm
I20240502 14:01:19.525135 31099 webserver.cc:460] Webserver started at http://127.16.217.8:40059/ using document root <none> and password file <none>
I20240502 14:01:19.525689 31099 fs_manager.cc:359] Metadata directory not provided
I20240502 14:01:19.525763 31099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:01:19.528939 31099 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:01:19.530884 31114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.531810 31099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:19.531905 31099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c59dc7abff9749abab221ce409d49f59"
format_stamp: "Formatted at 2024-05-02 14:01:09 on dist-test-slave-7msf"
I20240502 14:01:19.532349 31099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:01:19.560464 31099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:01:19.561190 31099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:01:19.561323 31099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:01:19.561841 31099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:01:19.562940 31099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:01:19.562992 31099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.563045 31099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:01:19.563068 31099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:01:19.573149 31099 rpc_server.cc:307] RPC server started. Bound to: 127.16.217.8:38573
I20240502 14:01:19.573199 31228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.217.8:38573 every 8 connection(s)
I20240502 14:01:19.574149 31099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:01:19.579053 17252 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhNHehp/build/debug/bin/kudu as pid 31099
I20240502 14:01:19.583976 31229 heartbeater.cc:347] Connected to a master server at 127.16.217.62:41117
I20240502 14:01:19.584232 31229 heartbeater.cc:464] Registering TS with master...
I20240502 14:01:19.584839 31229 heartbeater.cc:511] Master 127.16.217.62:41117 requested a full tablet report, sending...
I20240502 14:01:19.586027 28879 ts_manager.cc:194] Re-registered known tserver with Master: c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:01:19.586817 28879 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.217.8:40267
I20240502 14:01:19.929983 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:19.933874 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:19.934186 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:19.945588 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:19.947036 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.949954 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.964727 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.965746 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.981886 30763 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.981892 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.989403 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:19.992909 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:19.997773 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:19.998764 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.026470 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.032346 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.062664 30830 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:20.080773 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.099190 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.102756 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.106972 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.111605 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.122560 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.122954 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.126722 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.140990 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.149111 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.151743 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.153132 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.153949 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.156807 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.164966 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.166456 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.168676 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.178382 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.182288 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.192910 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.208930 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.217275 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.219239 30963 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:20.221484 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.228483 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.410317 31096 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:20.447953 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.455317 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.466578 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.469484 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.474761 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.487212 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.497282 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.499636 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.505359 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.527824 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.538697 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.552949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.557382 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.565491 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.566403 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.573354 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.573946 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.585744 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.589280 31229 heartbeater.cc:503] Master 127.16.217.62:41117 was elected leader, sending a full tablet report...
I20240502 14:01:20.590140 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.592824 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.600905 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.604110 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.604547 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.605787 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.611214 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.625509 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.645391 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.648892 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.651717 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.664355 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.667024 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.670960 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.679564 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.684726 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.691206 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.701649 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.706277 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.719492 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:20.733305 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.754513 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.878583 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.888655 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.901576 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.911844 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.936906 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.938261 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:20.941581 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.946408 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.953022 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.964392 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:20.980556 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:20.992365 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.992765 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:20.996379 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.001044 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.005702 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.006063 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.010464 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.018743 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.040781 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.051523 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:21.067924 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.077435 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.086747 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:21.089892 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.101871 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.105607 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.111672 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.112627 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.123106 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:21.125258 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:21.126677 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.129930 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.140192 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.152479 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.152983 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:21.173964 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.177745 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.183658 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:21.188581 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.246733 29291 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:21.248854 29028 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.251360 31988 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.252055 29161 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.253211 30255 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.259084 31989 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.261551 29028 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.263100 28875 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.263131 29161 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
W20240502 14:01:21.264335 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.290750 29161 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.291802 29028 consensus_queue.cc:237] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.292353 29424 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.292747 31742 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.293419 29291 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.293921 31479 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.294675 29161 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.294683 29293 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.295171 32054 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.295311 31479 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.297348 29424 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.297556 32054 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.299782 29159 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.299638 29425 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.300211 31893 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.300319 28875 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.303596 29293 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.309885 29028 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:21.310163 29359 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05a1a0bf1314d098f2bf65df58222e2. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:21.310251 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 785828dee4594ca8813b9418cc5dba2d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.310393 31893 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.312211 29095 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfd44eeeec934413a5001673430e6500. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.312894 31962 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.314607 29161 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.315766 28875 catalog_manager.cc:5484] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.317410 29293 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.317831 29292 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.319814 29161 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.320187 31989 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.320849 29425 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.321208 31989 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.323154 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a75497cf22014c0cb9d079b448284b3d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.323782 31907 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.324182 29425 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.324906 31893 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.325881 29161 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.328989 29161 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.329298 28875 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.330114 28875 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.333837 29028 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.350389 29293 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.352638 29161 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:21.352766 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.353204 31907 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.353231 29028 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:21.353413 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.353746 31907 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.357215 31907 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.359268 29161 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.360497 29293 consensus_queue.cc:237] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.360986 29028 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.361073 28878 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.363829 29425 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.364202 30255 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.367197 29161 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.369685 29425 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.370841 29293 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.371829 29028 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.372018 29161 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:21.372370 29359 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05a1a0bf1314d098f2bf65df58222e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.372637 31893 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.372838 29425 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.372860 32082 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.373214 29161 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.374217 30255 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.374503 30255 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.375344 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f05a1a0bf1314d098f2bf65df58222e2. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:21.375360 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1adebdbd104796b927963e86a15295. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.375253 31893 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
W20240502 14:01:21.375456 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.375633 30255 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.376673 29028 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.377557 32141 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.377712 29159 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.378698 28875 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.380486 29424 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.381467 29159 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.381738 28875 catalog_manager.cc:5484] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.397217 30255 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.397549 29161 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.400895 28883 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.400871 29425 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.416337 29293 consensus_queue.cc:237] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:21.417904 29425 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.418277 31988 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.418845 29161 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:21.420190 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1adebdbd104796b927963e86a15295. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:21.420413 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b1adebdbd104796b927963e86a15295. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.420508 31988 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.422153 31988 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.424734 29028 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.424785 29425 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.425946 29027 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:21.427553 28883 catalog_manager.cc:5484] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.427968 29161 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.428524 29424 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.428951 31479 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.429054 29425 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.429508 31479 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.430441 29161 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.431010 32057 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.431164 31479 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.431388 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88f85f143d845a1b7066244c606f8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.431802 29161 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
W20240502 14:01:21.434481 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b2d5d0f4044106bed079c3faa72447. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.435963 31962 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.436038 31479 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.438686 29425 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.438694 29160 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.438910 28882 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.439136 28882 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.440003 29424 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.441056 29159 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.474121 29027 consensus_queue.cc:237] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.475893 29424 raft_consensus.cc:1261] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.477310 32218 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.477715 29293 raft_consensus.cc:1261] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.478283 29292 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.478852 32218 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.480374 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.480393 29159 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:21.480803 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.480903 31988 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:21.481184 29424 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
W20240502 14:01:21.483088 29359 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed713f89a4fd47d79f03711dd14c1334. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.483439 29028 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:21.483770 28962 consensus_peers.cc:485] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc536794ffb453495d025f29ea152bc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.483894 32083 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.484412 29027 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:21.484879 31988 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:21.485560 29159 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.485920 32083 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.489365 32083 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
W20240502 14:01:21.489660 29227 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1b37b67e104154b6755c1762f8f6ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.491510 29028 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.493577 29159 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.494266 28879 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.498996 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.501133 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.505187 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.508796 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:21.521235 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.523567 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:21.530483 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.538343 32218 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.540247 29292 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.545079 28884 catalog_manager.cc:5484] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.545200 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.545125 29425 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.574443 29028 consensus_queue.cc:237] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
W20240502 14:01:21.576572 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19784717201a464c84607d2a43e651c2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.581118 29159 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.581936 31962 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.586174 29292 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.587029 31962 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.596666 32057 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.597185 29292 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.599798 29159 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.599923 28876 catalog_manager.cc:5484] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.611076 29160 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.613004 29292 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.613759 29425 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.613976 32054 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.614301 31306 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.618008 29093 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 015b587864d54da8aee36c3afd968e86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.623207 31306 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.625360 29292 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.626142 29425 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.629233 28884 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.642244 29292 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
W20240502 14:01:21.643973 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.643976 29160 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.644054 29425 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.644438 32141 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.645071 32141 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:01:21.645408 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.647221 31989 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.647626 29424 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.649928 29160 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.650068 28878 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.659309 29160 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.661499 29292 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.662184 29424 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.662727 29028 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.663092 31742 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.663390 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0176fb1a25a94c1c9d8cf0dc63577e91. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.663532 32054 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.664095 29424 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.664415 31907 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.664497 29028 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.664892 31907 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.667831 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dda4193aaaf459fb5049ff49541229a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.669898 32407 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.670250 29425 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.672633 29027 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.672641 28876 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.677927 31989 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.681094 29425 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.683602 29027 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.684123 28882 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.703192 29027 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.705354 29425 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.707119 29292 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.707451 31479 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.707789 31479 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.712347 32217 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
W20240502 14:01:21.712869 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a91fd318f0844c1961f16823a7ccc88. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.714385 29425 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.714928 28879 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.716153 29292 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:21.741837 29160 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:21.742506 32486 ts_tablet_manager.cc:902] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:21.743322 32486 tablet_copy_client.cc:308] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:21.745111 29292 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.745169 29425 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:21.745564 31742 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:21.745870 31742 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:01:21.746028 29093 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 015b587864d54da8aee36c3afd968e86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.749974 31742 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.751295 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.752319 29292 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.752684 28878 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:21.755810 29425 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
W20240502 14:01:21.757917 29094 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 015b587864d54da8aee36c3afd968e86. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.762907 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.764115 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.773633 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:21.777709 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:21.780136 32504 ts_tablet_manager.cc:902] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:21.780678 32504 tablet_copy_client.cc:308] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:21.789193 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:21.790205 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:21.791024 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:21.797235 32513 ts_tablet_manager.cc:902] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:21.797333 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 785828dee4594ca8813b9418cc5dba2d from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:49255)
I20240502 14:01:21.797489 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 785828dee4594ca8813b9418cc5dba2d from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:49255: session id = 12348ca540074b32912fdd6333484e0b-785828dee4594ca8813b9418cc5dba2d
I20240502 14:01:21.798326 32513 tablet_copy_client.cc:308] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:21.799111 29048 tablet_copy_source_session.cc:205] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.803840 32486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785828dee4594ca8813b9418cc5dba2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.805390 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet a75497cf22014c0cb9d079b448284b3d from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:59889)
I20240502 14:01:21.805547 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet a75497cf22014c0cb9d079b448284b3d from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:59889: session id = 12348ca540074b32912fdd6333484e0b-a75497cf22014c0cb9d079b448284b3d
I20240502 14:01:21.806807 29313 tablet_copy_source_session.cc:205] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.807292 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet f05a1a0bf1314d098f2bf65df58222e2 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:51685)
I20240502 14:01:21.807420 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet f05a1a0bf1314d098f2bf65df58222e2 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:51685: session id = 33ebf308f6f24e6e8768ef5f14ddc785-f05a1a0bf1314d098f2bf65df58222e2
I20240502 14:01:21.807884 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.808677 29448 tablet_copy_source_session.cc:205] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.811895 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.818688 32513 tablet_copy_client.cc:783] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.818915 32513 tablet_copy_client.cc:647] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.821754 32486 tablet_copy_client.cc:783] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.821926 32486 tablet_copy_client.cc:647] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.823544 32486 tablet_copy_client.cc:523] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.825021 32504 tablet_copy_client.cc:783] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.825170 32504 tablet_copy_client.cc:647] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.826324 32524 ts_tablet_manager.cc:902] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:21.826633 29292 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.826910 32513 tablet_copy_client.cc:523] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:01:21.829017 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.829677 32513 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:21.830008 29160 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:21.830477 32141 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:21.831648 32530 ts_tablet_manager.cc:902] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:21.832080 32524 tablet_copy_client.cc:308] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:21.832222 32530 tablet_copy_client.cc:308] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
W20240502 14:01:21.832316 29227 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.833822 29425 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:21.834190 32141 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:21.834519 32504 tablet_copy_client.cc:523] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:01:21.836093 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff65ce0568514e89894ec845fbda31f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.836405 32504 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:21.836829 32486 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:21.840993 32513 log.cc:824] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:21.841183 29160 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.842550 29425 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.842694 29027 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.843159 32407 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.843415 32054 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:21.843706 29095 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86cfc7d5709f414aac098ae3c05a48a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.845667 32504 log.cc:824] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:21.849833 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet f05a1a0bf1314d098f2bf65df58222e2 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:46805)
I20240502 14:01:21.849922 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet f05a1a0bf1314d098f2bf65df58222e2 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:46805: session id = c59dc7abff9749abab221ce409d49f59-f05a1a0bf1314d098f2bf65df58222e2
I20240502 14:01:21.850611 29448 tablet_copy_source_session.cc:205] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.854368 32524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.862886 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet cfd44eeeec934413a5001673430e6500 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:60507)
I20240502 14:01:21.863045 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet cfd44eeeec934413a5001673430e6500 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:60507: session id = c59dc7abff9749abab221ce409d49f59-cfd44eeeec934413a5001673430e6500
I20240502 14:01:21.864408 29181 tablet_copy_source_session.cc:205] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.866165 32530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd44eeeec934413a5001673430e6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.867107 32407 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.869282 29027 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.870546 32513 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.870870 32513 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:21.871181 32513 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.001s
I20240502 14:01:21.871762 28878 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.873060 29425 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.873708 32513 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.873940 32513 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.875026 32513 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.875947 32513 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:01:21.876176 32530 tablet_copy_client.cc:783] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.878302 32530 tablet_copy_client.cc:647] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.878427 32524 tablet_copy_client.cc:783] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.878719 32524 tablet_copy_client.cc:647] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.879168 32486 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.879593 32486 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:21.879997 32486 ts_tablet_manager.cc:1367] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.043s user 0.030s sys 0.000s
I20240502 14:01:21.881773 32524 tablet_copy_client.cc:523] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.882933 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-f05a1a0bf1314d098f2bf65df58222e2 received from {username='slave'} at 127.16.217.7:51685
I20240502 14:01:21.883030 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-f05a1a0bf1314d098f2bf65df58222e2 on tablet f05a1a0bf1314d098f2bf65df58222e2 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:21.882968 32486 raft_consensus.cc:348] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.883204 32486 raft_consensus.cc:729] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.883854 32486 consensus_queue.cc:260] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.884867 32486 ts_tablet_manager.cc:1398] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:01:21.885565 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-785828dee4594ca8813b9418cc5dba2d received from {username='slave'} at 127.16.217.6:49255
I20240502 14:01:21.885640 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 12348ca540074b32912fdd6333484e0b-785828dee4594ca8813b9418cc5dba2d on tablet 785828dee4594ca8813b9418cc5dba2d with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:21.886195 32549 ts_tablet_manager.cc:902] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:21.886771 32549 tablet_copy_client.cc:308] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:21.887293 32530 tablet_copy_client.cc:523] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.890782 32530 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:21.895255 29292 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.896837 29160 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:21.897035 29225 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400da91cbe234c79859b23ebe1643eb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.897218 32141 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.897415 29027 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.897783 32141 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.894969 32524 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:21.902102 32530 log.cc:824] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:21.908601 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet dcc536794ffb453495d025f29ea152bc from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:51565)
I20240502 14:01:21.908706 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet dcc536794ffb453495d025f29ea152bc from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:51565: session id = c59dc7abff9749abab221ce409d49f59-dcc536794ffb453495d025f29ea152bc
I20240502 14:01:21.909713 29048 tablet_copy_source_session.cc:205] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.910302 32566 ts_tablet_manager.cc:902] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:21.910665 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc536794ffb453495d025f29ea152bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.911075 32566 tablet_copy_client.cc:308] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:21.912778 32549 tablet_copy_client.cc:783] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.912902 32549 tablet_copy_client.cc:647] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.914309 32549 tablet_copy_client.cc:523] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.915513 32549 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:21.921998 32504 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.922358 32504 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:21.922521 32504 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.086s user 0.034s sys 0.000s
I20240502 14:01:21.923082 32504 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.923205 32504 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.923411 32504 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:21.924046 32504 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:21.924633 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-a75497cf22014c0cb9d079b448284b3d received from {username='slave'} at 127.16.217.6:59889
I20240502 14:01:21.924733 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 12348ca540074b32912fdd6333484e0b-a75497cf22014c0cb9d079b448284b3d on tablet a75497cf22014c0cb9d079b448284b3d with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:21.926713 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 5e1b37b67e104154b6755c1762f8f6ff from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:50553)
I20240502 14:01:21.926823 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 5e1b37b67e104154b6755c1762f8f6ff from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:50553: session id = 22dbbf207ba0440392e71a10c2ca78ca-5e1b37b67e104154b6755c1762f8f6ff
I20240502 14:01:21.927960 29313 tablet_copy_source_session.cc:205] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.930797 29159 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.931643 32566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.932559 29425 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:21.932981 31306 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.934021 29292 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:21.934458 29095 consensus_peers.cc:485] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15513e220e144883b216c4f3ac8ade0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:21.934679 31306 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:21.938922 32530 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.939237 32530 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:21.939525 32530 ts_tablet_manager.cc:1367] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.049s user 0.035s sys 0.001s
I20240502 14:01:21.940883 32566 tablet_copy_client.cc:783] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.941098 32566 tablet_copy_client.cc:647] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.941877 32530 raft_consensus.cc:348] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.942106 32530 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.942550 32566 tablet_copy_client.cc:523] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.942616 32530 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.944564 32530 ts_tablet_manager.cc:1398] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 14:01:21.944749 32566 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:21.948192 32549 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.948482 32549 tablet_bootstrap.cc:492] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:21.948657 32549 ts_tablet_manager.cc:1367] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.003s
I20240502 14:01:21.949128 32549 raft_consensus.cc:348] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.949215 32549 raft_consensus.cc:729] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.949333 32549 consensus_queue.cc:260] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.949633 32549 ts_tablet_manager.cc:1398] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:21.950168 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-dcc536794ffb453495d025f29ea152bc received from {username='slave'} at 127.16.217.8:51565
I20240502 14:01:21.950258 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session c59dc7abff9749abab221ce409d49f59-dcc536794ffb453495d025f29ea152bc on tablet dcc536794ffb453495d025f29ea152bc with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:21.954872 32566 log.cc:824] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Log is configured to *not* fsync() on all Append() calls
I20240502 14:01:21.956781 31306 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.958201 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-cfd44eeeec934413a5001673430e6500 received from {username='slave'} at 127.16.217.8:60507
I20240502 14:01:21.958288 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session c59dc7abff9749abab221ce409d49f59-cfd44eeeec934413a5001673430e6500 on tablet cfd44eeeec934413a5001673430e6500 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:21.958341 32141 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.960662 29292 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.960844 29425 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.962162 32530 ts_tablet_manager.cc:902] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:21.962551 32530 tablet_copy_client.cc:308] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:21.963879 29027 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.965389 28883 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.965531 32549 ts_tablet_manager.cc:902] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:21.965970 32549 tablet_copy_client.cc:308] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:21.966569 29160 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:21.966709 28878 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:21.969697 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet ed713f89a4fd47d79f03711dd14c1334 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:46805)
I20240502 14:01:21.969794 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet ed713f89a4fd47d79f03711dd14c1334 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:46805: session id = c59dc7abff9749abab221ce409d49f59-ed713f89a4fd47d79f03711dd14c1334
I20240502 14:01:21.970625 29448 tablet_copy_source_session.cc:205] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.973359 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed713f89a4fd47d79f03711dd14c1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.975368 32549 tablet_copy_client.cc:783] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.975507 32549 tablet_copy_client.cc:647] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.978893 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 5e1b37b67e104154b6755c1762f8f6ff from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:53621)
I20240502 14:01:21.978992 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 5e1b37b67e104154b6755c1762f8f6ff from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:53621: session id = c59dc7abff9749abab221ce409d49f59-5e1b37b67e104154b6755c1762f8f6ff
I20240502 14:01:21.979979 29313 tablet_copy_source_session.cc:205] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.980834 32530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.981392 32524 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:21.981683 32524 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:21.981894 32524 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.087s user 0.023s sys 0.004s
I20240502 14:01:21.982350 32524 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.982452 32524 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:21.982590 32524 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:21.982760 32530 tablet_copy_client.cc:783] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.982918 32530 tablet_copy_client.cc:647] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.983007 32524 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:21.983412 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-f05a1a0bf1314d098f2bf65df58222e2 received from {username='slave'} at 127.16.217.8:46805
I20240502 14:01:21.983510 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session c59dc7abff9749abab221ce409d49f59-f05a1a0bf1314d098f2bf65df58222e2 on tablet f05a1a0bf1314d098f2bf65df58222e2 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:21.984723 32530 tablet_copy_client.cc:523] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.985890 32530 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:21.986461 32613 ts_tablet_manager.cc:902] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:21.987075 32613 tablet_copy_client.cc:308] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:21.991199 32549 tablet_copy_client.cc:523] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.992020 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 81b2d5d0f4044106bed079c3faa72447 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:21.992110 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 81b2d5d0f4044106bed079c3faa72447 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-81b2d5d0f4044106bed079c3faa72447
I20240502 14:01:21.992364 32549 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:21.993094 29048 tablet_copy_source_session.cc:205] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:21.994443 32613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:21.997262 32613 tablet_copy_client.cc:783] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:21.997447 32613 tablet_copy_client.cc:647] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:21.998445 32635 ts_tablet_manager.cc:902] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:21.998478 32613 tablet_copy_client.cc:523] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:21.998999 32635 tablet_copy_client.cc:308] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:21.999482 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 2b1adebdbd104796b927963e86a15295 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:50553)
I20240502 14:01:21.999580 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 2b1adebdbd104796b927963e86a15295 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:50553: session id = 22dbbf207ba0440392e71a10c2ca78ca-2b1adebdbd104796b927963e86a15295
I20240502 14:01:22.000474 29313 tablet_copy_source_session.cc:205] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.001245 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1adebdbd104796b927963e86a15295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.004462 32613 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:22.012348 29027 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.012976 32635 tablet_copy_client.cc:783] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.013175 32635 tablet_copy_client.cc:647] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.014433 29292 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:22.014534 28962 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ffcf19e66e4adca28d87cbe2245964. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.014962 29425 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.015208 32530 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.015631 32530 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.015633 32218 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.015800 32530 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.030s user 0.027s sys 0.000s
I20240502 14:01:22.016530 32530 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.017021 32530 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.017136 32530 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.017261 32513 ts_tablet_manager.cc:902] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.017393 32530 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:22.017690 32513 tablet_copy_client.cc:308] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.018771 32218 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.021466 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-5e1b37b67e104154b6755c1762f8f6ff received from {username='slave'} at 127.16.217.8:53621
I20240502 14:01:22.021565 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session c59dc7abff9749abab221ce409d49f59-5e1b37b67e104154b6755c1762f8f6ff on tablet 5e1b37b67e104154b6755c1762f8f6ff with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:22.024981 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 2b1adebdbd104796b927963e86a15295 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:53421)
I20240502 14:01:22.025084 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 2b1adebdbd104796b927963e86a15295 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:53421: session id = 33ebf308f6f24e6e8768ef5f14ddc785-2b1adebdbd104796b927963e86a15295
I20240502 14:01:22.026017 29313 tablet_copy_source_session.cc:205] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.026405 32635 tablet_copy_client.cc:523] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.026813 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1adebdbd104796b927963e86a15295. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.027865 32549 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.028126 32549 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.028297 32549 ts_tablet_manager.cc:1367] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.000s
I20240502 14:01:22.028721 32549 raft_consensus.cc:348] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.028825 32549 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.028944 32549 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.029271 32549 ts_tablet_manager.cc:1398] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.029489 32513 tablet_copy_client.cc:783] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.029690 32513 tablet_copy_client.cc:647] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.029680 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-ed713f89a4fd47d79f03711dd14c1334 received from {username='slave'} at 127.16.217.8:46805
I20240502 14:01:22.029752 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session c59dc7abff9749abab221ce409d49f59-ed713f89a4fd47d79f03711dd14c1334 on tablet ed713f89a4fd47d79f03711dd14c1334 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:22.030506 32504 ts_tablet_manager.cc:902] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.030897 32504 tablet_copy_client.cc:308] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.032052 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 5e1b37b67e104154b6755c1762f8f6ff from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:59889)
I20240502 14:01:22.032145 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 5e1b37b67e104154b6755c1762f8f6ff from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:59889: session id = 12348ca540074b32912fdd6333484e0b-5e1b37b67e104154b6755c1762f8f6ff
I20240502 14:01:22.033036 29313 tablet_copy_source_session.cc:205] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.033993 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1b37b67e104154b6755c1762f8f6ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.036161 32504 tablet_copy_client.cc:783] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.036841 32504 tablet_copy_client.cc:647] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.038167 32635 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:22.038818 32566 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.039258 32566 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.039609 32566 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.095s user 0.032s sys 0.000s
I20240502 14:01:22.040407 32486 ts_tablet_manager.cc:902] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.040860 32486 tablet_copy_client.cc:308] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.041407 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet c88f85f143d845a1b7066244c606f8ba from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:49255)
I20240502 14:01:22.041518 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet c88f85f143d845a1b7066244c606f8ba from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:49255: session id = 12348ca540074b32912fdd6333484e0b-c88f85f143d845a1b7066244c606f8ba
I20240502 14:01:22.042457 29048 tablet_copy_source_session.cc:205] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.043193 32486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.043098 32566 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.043761 32566 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.044097 32566 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.044430 32513 tablet_copy_client.cc:523] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.045001 32566 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240502 14:01:22.045122 32486 tablet_copy_client.cc:783] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.045243 32486 tablet_copy_client.cc:647] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.045718 32513 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:22.048777 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-5e1b37b67e104154b6755c1762f8f6ff received from {username='slave'} at 127.16.217.5:50553
I20240502 14:01:22.048854 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-5e1b37b67e104154b6755c1762f8f6ff on tablet 5e1b37b67e104154b6755c1762f8f6ff with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.058182 32218 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.060669 29425 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.061424 28884 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.062270 32486 tablet_copy_client.cc:523] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.063622 32486 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.065953 29292 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.072348 32504 tablet_copy_client.cc:523] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.073642 32504 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.094583 32613 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.094956 32613 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.095165 32613 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.091s user 0.026s sys 0.000s
I20240502 14:01:22.095795 32613 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.095957 32613 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.096127 32613 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.096503 32613 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.097443 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-81b2d5d0f4044106bed079c3faa72447 received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:22.097533 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-81b2d5d0f4044106bed079c3faa72447 on tablet 81b2d5d0f4044106bed079c3faa72447 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.107309 32613 ts_tablet_manager.cc:902] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.107728 32613 tablet_copy_client.cc:308] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.109127 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 19784717201a464c84607d2a43e651c2 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:22.109236 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 19784717201a464c84607d2a43e651c2 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-19784717201a464c84607d2a43e651c2
I20240502 14:01:22.110145 29048 tablet_copy_source_session.cc:205] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.112709 32613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19784717201a464c84607d2a43e651c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.113860 32717 ts_tablet_manager.cc:902] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.114146 32486 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.114398 32486 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:22.114634 32486 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.051s user 0.028s sys 0.000s
I20240502 14:01:22.115069 32717 tablet_copy_client.cc:308] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.115173 32486 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.115295 32486 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.115453 32486 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.116052 32613 tablet_copy_client.cc:783] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.116211 32613 tablet_copy_client.cc:647] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.117623 32613 tablet_copy_client.cc:523] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.118840 32613 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:22.120901 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 1dda4193aaaf459fb5049ff49541229a from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:51559)
I20240502 14:01:22.121006 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 1dda4193aaaf459fb5049ff49541229a from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:51559: session id = 33ebf308f6f24e6e8768ef5f14ddc785-1dda4193aaaf459fb5049ff49541229a
I20240502 14:01:22.121829 32513 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.121898 29181 tablet_copy_source_session.cc:205] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.122153 32513 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.122335 32513 ts_tablet_manager.cc:1367] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.077s user 0.027s sys 0.001s
I20240502 14:01:22.122699 32717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.123055 32513 raft_consensus.cc:348] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.123217 32513 raft_consensus.cc:729] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.123355 32486 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:01:22.123400 32513 consensus_queue.cc:260] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.124408 32513 ts_tablet_manager.cc:1398] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:22.124888 32722 ts_tablet_manager.cc:902] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.124913 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-2b1adebdbd104796b927963e86a15295 received from {username='slave'} at 127.16.217.7:53421
I20240502 14:01:22.125013 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-2b1adebdbd104796b927963e86a15295 on tablet 2b1adebdbd104796b927963e86a15295 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.125449 32722 tablet_copy_client.cc:308] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.125926 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:53421)
I20240502 14:01:22.126008 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 0176fb1a25a94c1c9d8cf0dc63577e91 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:53421: session id = 33ebf308f6f24e6e8768ef5f14ddc785-0176fb1a25a94c1c9d8cf0dc63577e91
I20240502 14:01:22.126806 29313 tablet_copy_source_session.cc:205] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.127270 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-c88f85f143d845a1b7066244c606f8ba received from {username='slave'} at 127.16.217.6:49255
I20240502 14:01:22.127348 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 12348ca540074b32912fdd6333484e0b-c88f85f143d845a1b7066244c606f8ba on tablet c88f85f143d845a1b7066244c606f8ba with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.127811 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0176fb1a25a94c1c9d8cf0dc63577e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.129570 32717 tablet_copy_client.cc:783] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.129705 32717 tablet_copy_client.cc:647] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.130336 32722 tablet_copy_client.cc:783] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.130496 32722 tablet_copy_client.cc:647] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.130652 32717 tablet_copy_client.cc:523] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.131669 32717 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:22.135120 32513 ts_tablet_manager.cc:902] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.135569 32513 tablet_copy_client.cc:308] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.139839 29160 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.140257 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 5a91fd318f0844c1961f16823a7ccc88 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:54097)
I20240502 14:01:22.140373 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 5a91fd318f0844c1961f16823a7ccc88 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:54097: session id = 33ebf308f6f24e6e8768ef5f14ddc785-5a91fd318f0844c1961f16823a7ccc88
I20240502 14:01:22.140901 32504 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.141218 32504 tablet_bootstrap.cc:492] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:22.141311 29048 tablet_copy_source_session.cc:205] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.141453 32504 ts_tablet_manager.cc:1367] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.068s user 0.024s sys 0.003s
W20240502 14:01:22.141686 29095 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86cfc7d5709f414aac098ae3c05a48a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.142026 29027 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.142187 32504 raft_consensus.cc:348] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.142344 29425 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.142693 32504 raft_consensus.cc:729] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.142815 32504 consensus_queue.cc:260] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.143198 32407 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.143762 32407 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.143942 32504 ts_tablet_manager.cc:1398] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:22.144330 32722 tablet_copy_client.cc:523] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.145144 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-5e1b37b67e104154b6755c1762f8f6ff received from {username='slave'} at 127.16.217.6:59889
I20240502 14:01:22.145224 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 12348ca540074b32912fdd6333484e0b-5e1b37b67e104154b6755c1762f8f6ff on tablet 5e1b37b67e104154b6755c1762f8f6ff with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.145485 32722 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
W20240502 14:01:22.146029 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86cfc7d5709f414aac098ae3c05a48a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.146081 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.147648 32407 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.148135 32513 tablet_copy_client.cc:783] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.148262 32513 tablet_copy_client.cc:647] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.149472 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.152448 29425 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.154808 28882 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.157737 29027 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.163475 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:22.163488 32613 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.163753 32613 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.163911 32613 ts_tablet_manager.cc:1367] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.000s
I20240502 14:01:22.164419 32613 raft_consensus.cc:348] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.164533 32613 raft_consensus.cc:729] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.164664 32613 consensus_queue.cc:260] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.164928 32613 ts_tablet_manager.cc:1398] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:22.165625 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-19784717201a464c84607d2a43e651c2 received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:22.165702 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-19784717201a464c84607d2a43e651c2 on tablet 19784717201a464c84607d2a43e651c2 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.171885 32635 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.172886 32635 tablet_bootstrap.cc:492] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.173070 32635 ts_tablet_manager.cc:1367] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.135s user 0.028s sys 0.000s
I20240502 14:01:22.173475 32635 raft_consensus.cc:348] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.173578 32635 raft_consensus.cc:729] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.173738 32635 consensus_queue.cc:260] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.173987 32635 ts_tablet_manager.cc:1398] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.174484 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-2b1adebdbd104796b927963e86a15295 received from {username='slave'} at 127.16.217.5:50553
I20240502 14:01:22.174554 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-2b1adebdbd104796b927963e86a15295 on tablet 2b1adebdbd104796b927963e86a15295 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.176498 32513 tablet_copy_client.cc:523] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.177569 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:22.177892 32513 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:22.182853 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.192581 32504 ts_tablet_manager.cc:902] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.192996 32504 tablet_copy_client.cc:308] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.197845 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:22.198621 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 015b587864d54da8aee36c3afd968e86 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:41057)
I20240502 14:01:22.198717 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 015b587864d54da8aee36c3afd968e86 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:41057: session id = 12348ca540074b32912fdd6333484e0b-015b587864d54da8aee36c3afd968e86
I20240502 14:01:22.199620 29181 tablet_copy_source_session.cc:205] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.200446 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015b587864d54da8aee36c3afd968e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.202492 32504 tablet_copy_client.cc:783] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.202633 32504 tablet_copy_client.cc:647] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.203389 32722 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.203662 32722 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.203806 32722 ts_tablet_manager.cc:1367] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.058s user 0.022s sys 0.000s
I20240502 14:01:22.204216 32722 raft_consensus.cc:348] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.204341 32722 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.204502 32722 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.205415 32722 ts_tablet_manager.cc:1398] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:01:22.205883 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-0176fb1a25a94c1c9d8cf0dc63577e91 received from {username='slave'} at 127.16.217.7:53421
I20240502 14:01:22.205966 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-0176fb1a25a94c1c9d8cf0dc63577e91 on tablet 0176fb1a25a94c1c9d8cf0dc63577e91 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.206822 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.209527 32717 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.210364 32717 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.210529 32717 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.079s user 0.024s sys 0.002s
I20240502 14:01:22.210880 32717 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.210966 32717 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.211086 32717 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.211694 32717 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.212081 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-1dda4193aaaf459fb5049ff49541229a received from {username='slave'} at 127.16.217.7:51559
I20240502 14:01:22.212163 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-1dda4193aaaf459fb5049ff49541229a on tablet 1dda4193aaaf459fb5049ff49541229a with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.221347 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.225513 32513 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.225849 32513 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.226464 32513 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.003s
I20240502 14:01:22.226922 32549 ts_tablet_manager.cc:902] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.226949 32513 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.227190 32513 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.227360 32549 tablet_copy_client.cc:308] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.227399 32513 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.227854 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet ff65ce0568514e89894ec845fbda31f3 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:53621)
I20240502 14:01:22.227949 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet ff65ce0568514e89894ec845fbda31f3 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:53621: session id = c59dc7abff9749abab221ce409d49f59-ff65ce0568514e89894ec845fbda31f3
I20240502 14:01:22.228941 29313 tablet_copy_source_session.cc:205] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.229395 32513 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:22.229789 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.231766 32549 tablet_copy_client.cc:783] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.231896 32549 tablet_copy_client.cc:647] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.232139 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-5a91fd318f0844c1961f16823a7ccc88 received from {username='slave'} at 127.16.217.7:54097
I20240502 14:01:22.232213 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-5a91fd318f0844c1961f16823a7ccc88 on tablet 5a91fd318f0844c1961f16823a7ccc88 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.239444 30916 raft_consensus.cc:1203] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.243356 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:22.244582 32504 tablet_copy_client.cc:523] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.246114 32504 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.248772 29027 consensus_queue.cc:237] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.250167 29425 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:22.250372 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db68071f838a4e209f3db69bea6e54f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.250519 32057 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.251068 29292 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.251452 32057 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.253957 32549 tablet_copy_client.cc:523] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.255256 32549 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:22.262220 32057 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.264209 29425 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.266562 29292 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.267517 28883 catalog_manager.cc:5484] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.274583 32486 ts_tablet_manager.cc:902] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.275070 32486 tablet_copy_client.cc:308] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.275506 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 400da91cbe234c79859b23ebe1643eb0 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:59889)
I20240502 14:01:22.275610 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 400da91cbe234c79859b23ebe1643eb0 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:59889: session id = 12348ca540074b32912fdd6333484e0b-400da91cbe234c79859b23ebe1643eb0
I20240502 14:01:22.276652 29313 tablet_copy_source_session.cc:205] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.277441 32486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400da91cbe234c79859b23ebe1643eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.279563 32486 tablet_copy_client.cc:783] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.279691 32486 tablet_copy_client.cc:647] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.279832 32504 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.280058 32504 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:22.280211 32504 ts_tablet_manager.cc:1367] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.000s
I20240502 14:01:22.280944 32504 raft_consensus.cc:348] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.281062 32504 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.281214 32504 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.281478 29160 consensus_queue.cc:237] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.281957 32504 ts_tablet_manager.cc:1398] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:22.282902 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-015b587864d54da8aee36c3afd968e86 received from {username='slave'} at 127.16.217.6:41057
I20240502 14:01:22.282970 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 12348ca540074b32912fdd6333484e0b-015b587864d54da8aee36c3afd968e86 on tablet 015b587864d54da8aee36c3afd968e86 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.283128 29027 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.283685 29292 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.284444 32407 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.284824 32407 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:22.285262 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6787832a36648aca61dd26a363089d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.286149 32635 ts_tablet_manager.cc:902] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.286612 32635 tablet_copy_client.cc:308] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.288800 32054 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.291641 29292 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.292804 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 015b587864d54da8aee36c3afd968e86 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:33903)
I20240502 14:01:22.292902 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 015b587864d54da8aee36c3afd968e86 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:33903: session id = 22dbbf207ba0440392e71a10c2ca78ca-015b587864d54da8aee36c3afd968e86
I20240502 14:01:22.293038 32513 ts_tablet_manager.cc:902] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.293175 32486 tablet_copy_client.cc:523] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.293691 29181 tablet_copy_source_session.cc:205] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.294052 32513 tablet_copy_client.cc:308] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.294454 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 015b587864d54da8aee36c3afd968e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.294443 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet ff65ce0568514e89894ec845fbda31f3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:53421)
I20240502 14:01:22.294476 32486 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.294523 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet ff65ce0568514e89894ec845fbda31f3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:53421: session id = 33ebf308f6f24e6e8768ef5f14ddc785-ff65ce0568514e89894ec845fbda31f3
I20240502 14:01:22.295367 29313 tablet_copy_source_session.cc:205] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.296505 32635 tablet_copy_client.cc:783] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.296674 32635 tablet_copy_client.cc:647] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.296818 28879 catalog_manager.cc:5484] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.298115 32635 tablet_copy_client.cc:523] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.298429 32549 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.298662 32549 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.298808 32549 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.044s user 0.019s sys 0.004s
I20240502 14:01:22.299197 32635 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:22.299234 32549 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.299451 32549 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.299556 32549 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.299841 32549 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:22.299950 32504 ts_tablet_manager.cc:902] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:22.300331 32504 tablet_copy_client.cc:308] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:22.300429 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.300750 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet ff65ce0568514e89894ec845fbda31f3 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:59889)
I20240502 14:01:22.300832 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet ff65ce0568514e89894ec845fbda31f3 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:59889: session id = 12348ca540074b32912fdd6333484e0b-ff65ce0568514e89894ec845fbda31f3
I20240502 14:01:22.301622 29312 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-ff65ce0568514e89894ec845fbda31f3 received from {username='slave'} at 127.16.217.8:53621
I20240502 14:01:22.301689 29312 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session c59dc7abff9749abab221ce409d49f59-ff65ce0568514e89894ec845fbda31f3 on tablet ff65ce0568514e89894ec845fbda31f3 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:22.301647 29313 tablet_copy_source_session.cc:205] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.302387 32513 tablet_copy_client.cc:783] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.302517 32513 tablet_copy_client.cc:647] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.302809 29027 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.303054 32504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff65ce0568514e89894ec845fbda31f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.305123 32504 tablet_copy_client.cc:783] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.305244 32504 tablet_copy_client.cc:647] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.311631 32504 tablet_copy_client.cc:523] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.312773 32504 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.317986 32513 tablet_copy_client.cc:523] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.319247 32513 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:22.330621 32635 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.330956 32635 tablet_bootstrap.cc:492] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.331144 32635 ts_tablet_manager.cc:1367] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.000s
I20240502 14:01:22.331683 32635 raft_consensus.cc:348] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.331933 32635 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.332130 32635 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.332561 32635 ts_tablet_manager.cc:1398] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.333026 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-015b587864d54da8aee36c3afd968e86 received from {username='slave'} at 127.16.217.5:33903
I20240502 14:01:22.333101 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-015b587864d54da8aee36c3afd968e86 on tablet 015b587864d54da8aee36c3afd968e86 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.335290 32486 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.335552 32486 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:22.335722 32486 ts_tablet_manager.cc:1367] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.000s
I20240502 14:01:22.336171 32486 raft_consensus.cc:348] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.336282 32486 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.336432 32486 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.336748 32486 ts_tablet_manager.cc:1398] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:01:22.337622 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-400da91cbe234c79859b23ebe1643eb0 received from {username='slave'} at 127.16.217.6:59889
I20240502 14:01:22.337715 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 12348ca540074b32912fdd6333484e0b-400da91cbe234c79859b23ebe1643eb0 on tablet 400da91cbe234c79859b23ebe1643eb0 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.340819 31047 raft_consensus.cc:1203] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.345890 32504 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.346151 32504 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:22.346320 32504 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.002s
I20240502 14:01:22.346730 32504 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.346969 32504 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.347054 32504 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.348908 32504 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:22.352125 32513 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 14:01:22.352663 32513 tablet_bootstrap.cc:492] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.352877 32513 ts_tablet_manager.cc:1367] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.000s
I20240502 14:01:22.353341 32513 raft_consensus.cc:348] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.353564 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-ff65ce0568514e89894ec845fbda31f3 received from {username='slave'} at 127.16.217.6:59889
I20240502 14:01:22.353638 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 12348ca540074b32912fdd6333484e0b-ff65ce0568514e89894ec845fbda31f3 on tablet ff65ce0568514e89894ec845fbda31f3 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.353873 32513 raft_consensus.cc:729] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.353998 32513 consensus_queue.cc:260] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.354465 32513 ts_tablet_manager.cc:1398] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:22.355057 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-ff65ce0568514e89894ec845fbda31f3 received from {username='slave'} at 127.16.217.7:53421
I20240502 14:01:22.355137 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-ff65ce0568514e89894ec845fbda31f3 on tablet ff65ce0568514e89894ec845fbda31f3 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.358655 31181 raft_consensus.cc:1203] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.363646 29027 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.364521 29028 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.365123 31047 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:22.365676 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a91fd318f0844c1961f16823a7ccc88. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.365971 29425 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.366372 29425 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.366528 29293 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:22.366590 28962 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ffcf19e66e4adca28d87cbe2245964. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.366523 29292 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.366751 31962 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.366885 32057 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.367007 32218 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.367091 32057 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.367157 31479 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:01:22.367642 28960 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ffcf19e66e4adca28d87cbe2245964. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.369016 29426 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.370395 32218 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.371003 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:22.371125 29358 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a476d1664de7456ab21ddbf2e12b4f0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.371228 29293 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.371291 376 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.371538 30571 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.371670 32217 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.372658 29293 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.373338 31181 raft_consensus.cc:1203] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.373282 28883 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.374800 29293 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.375665 29425 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.375651 29426 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.378134 28879 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.378177 31047 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.382064 376 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.383359 30916 raft_consensus.cc:1203] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.383294 29292 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.387295 29160 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.389896 28883 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.394475 31181 raft_consensus.cc:1203] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.396610 30255 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.397115 31181 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.398519 29160 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.400065 29028 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.402223 28878 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.417030 30783 raft_consensus.cc:1203] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.429463 29425 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.431087 29292 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.431497 377 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.432662 29160 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.434147 377 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:22.436097 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b1230a7230f48558da1069a5988b981. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.440428 377 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.442869 29160 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.445689 29292 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.446362 28884 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.455585 31181 raft_consensus.cc:1203] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.461371 30783 raft_consensus.cc:1203] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.461712 30783 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.473488 31047 raft_consensus.cc:1203] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.478401 31181 raft_consensus.cc:1203] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.504540 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.512907 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.514117 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:22.518522 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.528805 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:22.536901 31047 raft_consensus.cc:1203] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.544880 32513 ts_tablet_manager.cc:902] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.545358 32513 tablet_copy_client.cc:308] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.545785 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:51559)
I20240502 14:01:22.545884 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:51559: session id = 33ebf308f6f24e6e8768ef5f14ddc785-86cfc7d5709f414aac098ae3c05a48a3
I20240502 14:01:22.546715 29181 tablet_copy_source_session.cc:205] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.547330 32549 ts_tablet_manager.cc:902] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.547547 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.547863 32549 tablet_copy_client.cc:308] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.548333 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 15513e220e144883b216c4f3ac8ade0b from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:60507)
I20240502 14:01:22.548414 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 15513e220e144883b216c4f3ac8ade0b from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:60507: session id = c59dc7abff9749abab221ce409d49f59-15513e220e144883b216c4f3ac8ade0b
I20240502 14:01:22.549232 29181 tablet_copy_source_session.cc:205] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.550087 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.550091 32513 tablet_copy_client.cc:783] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.550216 32513 tablet_copy_client.cc:647] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.552163 32549 tablet_copy_client.cc:783] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.553879 29028 consensus_queue.cc:237] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.555032 32513 tablet_copy_client.cc:523] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.555296 29292 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.555685 31962 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.556350 32513 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
W20240502 14:01:22.556680 28962 consensus_peers.cc:485] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 252f4087c0e5443093e856fe676839b5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.557654 29160 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.558043 32218 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.559871 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:22.560405 32549 tablet_copy_client.cc:647] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.562806 32549 tablet_copy_client.cc:523] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.563695 30916 raft_consensus.cc:1203] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.565790 32549 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:22.571887 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.572086 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.580886 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:22.600855 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.605149 30783 raft_consensus.cc:1203] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.610792 31181 raft_consensus.cc:1203] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.611243 559 ts_tablet_manager.cc:902] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.613124 559 tablet_copy_client.cc:308] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.613091 30255 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.613611 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:60507)
I20240502 14:01:22.613720 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:60507: session id = c59dc7abff9749abab221ce409d49f59-86cfc7d5709f414aac098ae3c05a48a3
I20240502 14:01:22.614701 29181 tablet_copy_source_session.cc:205] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.615537 559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.617609 559 tablet_copy_client.cc:783] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.617789 559 tablet_copy_client.cc:647] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.617746 31178 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.618752 559 tablet_copy_client.cc:523] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.619808 559 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:22.620075 28879 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.622429 29425 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.625434 29160 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.627058 30783 raft_consensus.cc:1203] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.630829 32513 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.631827 32513 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.632110 32513 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.076s user 0.025s sys 0.000s
I20240502 14:01:22.632712 32513 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.632825 32513 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.632982 32513 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.634783 32513 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:22.635752 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.636547 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-86cfc7d5709f414aac098ae3c05a48a3 received from {username='slave'} at 127.16.217.7:51559
I20240502 14:01:22.636617 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-86cfc7d5709f414aac098ae3c05a48a3 on tablet 86cfc7d5709f414aac098ae3c05a48a3 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.638247 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:22.638275 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.647891 30916 raft_consensus.cc:1203] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.648696 30916 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.663798 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:22.665306 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.667135 32218 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.670117 29160 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.673079 28879 catalog_manager.cc:5484] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.673087 29292 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.673260 559 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.673527 559 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.673674 559 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.000s
I20240502 14:01:22.674089 559 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.674193 559 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.674443 559 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.674717 559 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:22.675135 31047 raft_consensus.cc:1203] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.675289 31047 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.676563 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-86cfc7d5709f414aac098ae3c05a48a3 received from {username='slave'} at 127.16.217.8:60507
I20240502 14:01:22.676626 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session c59dc7abff9749abab221ce409d49f59-86cfc7d5709f414aac098ae3c05a48a3 on tablet 86cfc7d5709f414aac098ae3c05a48a3 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:22.681026 30916 raft_consensus.cc:1203] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.692143 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:22.694119 30916 raft_consensus.cc:1203] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:22.694263 30916 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.698812 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:22.707540 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:22.718030 31047 raft_consensus.cc:1203] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.725183 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.735025 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.736486 32513 ts_tablet_manager.cc:902] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.736863 32513 tablet_copy_client.cc:308] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.737291 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet db68071f838a4e209f3db69bea6e54f3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:54097)
I20240502 14:01:22.737393 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet db68071f838a4e209f3db69bea6e54f3 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:54097: session id = 33ebf308f6f24e6e8768ef5f14ddc785-db68071f838a4e209f3db69bea6e54f3
I20240502 14:01:22.738273 29048 tablet_copy_source_session.cc:205] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.738943 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:22.739046 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db68071f838a4e209f3db69bea6e54f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.741109 32513 tablet_copy_client.cc:783] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.741240 32513 tablet_copy_client.cc:647] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.742841 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:22.745536 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:22.750322 32635 ts_tablet_manager.cc:902] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:22.750728 32635 tablet_copy_client.cc:308] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:22.750934 32549 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.751199 32549 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.751348 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet e6787832a36648aca61dd26a363089d5 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:33903)
I20240502 14:01:22.751459 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet e6787832a36648aca61dd26a363089d5 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:33903: session id = 22dbbf207ba0440392e71a10c2ca78ca-e6787832a36648aca61dd26a363089d5
I20240502 14:01:22.752396 29181 tablet_copy_source_session.cc:205] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.752887 32549 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.188s user 0.021s sys 0.003s
I20240502 14:01:22.753335 32549 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.753454 32549 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.753592 32549 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.753854 32549 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:22.753963 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6787832a36648aca61dd26a363089d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.754196 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-15513e220e144883b216c4f3ac8ade0b received from {username='slave'} at 127.16.217.8:60507
I20240502 14:01:22.754254 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session c59dc7abff9749abab221ce409d49f59-15513e220e144883b216c4f3ac8ade0b on tablet 15513e220e144883b216c4f3ac8ade0b with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:22.755936 32635 tablet_copy_client.cc:783] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.756086 32635 tablet_copy_client.cc:647] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.756877 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:22.760380 32513 tablet_copy_client.cc:523] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.761790 32513 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:22.777587 32635 tablet_copy_client.cc:523] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.779115 32635 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:22.789270 29160 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.791016 29028 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.791831 32407 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:22.793130 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa944cd0b82429da02bd3dd35e0cce7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.793387 29425 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.793793 32407 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.812249 32635 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.812551 32635 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:22.812749 32635 ts_tablet_manager.cc:1367] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.009s
I20240502 14:01:22.813160 32635 raft_consensus.cc:348] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.813283 32635 raft_consensus.cc:729] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.813398 32635 consensus_queue.cc:260] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.813712 32635 ts_tablet_manager.cc:1398] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.814105 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-e6787832a36648aca61dd26a363089d5 received from {username='slave'} at 127.16.217.5:33903
I20240502 14:01:22.814184 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-e6787832a36648aca61dd26a363089d5 on tablet e6787832a36648aca61dd26a363089d5 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:22.816350 31742 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.818429 29028 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.818943 29425 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.821173 28881 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index -1 to 4, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.822180 32513 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.822443 32513 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:22.822607 32513 ts_tablet_manager.cc:1367] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.061s user 0.023s sys 0.000s
I20240502 14:01:22.823017 32513 raft_consensus.cc:348] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.823117 32513 raft_consensus.cc:729] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.823262 32513 consensus_queue.cc:260] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.823757 32513 ts_tablet_manager.cc:1398] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:22.824489 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-db68071f838a4e209f3db69bea6e54f3 received from {username='slave'} at 127.16.217.7:54097
I20240502 14:01:22.824570 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-db68071f838a4e209f3db69bea6e54f3 on tablet db68071f838a4e209f3db69bea6e54f3 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:22.827179 29028 consensus_queue.cc:237] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.829144 29425 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.829414 29292 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.829432 31047 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.829530 31962 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.829762 378 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:01:22.830638 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db68071f838a4e209f3db69bea6e54f3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.830670 378 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.832314 378 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.833669 31047 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.835033 29292 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.836146 28882 catalog_manager.cc:5484] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 reported cstate change: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.837841 29425 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.846316 32549 ts_tablet_manager.cc:902] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.846769 32549 tablet_copy_client.cc:308] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.847203 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet d5ffcf19e66e4adca28d87cbe2245964 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:51565)
I20240502 14:01:22.847307 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet d5ffcf19e66e4adca28d87cbe2245964 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:51565: session id = c59dc7abff9749abab221ce409d49f59-d5ffcf19e66e4adca28d87cbe2245964
I20240502 14:01:22.848362 29048 tablet_copy_source_session.cc:205] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.849280 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ffcf19e66e4adca28d87cbe2245964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.851850 32549 tablet_copy_client.cc:783] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.852020 32549 tablet_copy_client.cc:647] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.853722 32549 tablet_copy_client.cc:523] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.855022 32549 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:22.858682 30783 raft_consensus.cc:1203] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:22.869051 29425 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.871306 29292 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.871793 382 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.872844 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:22.873354 29358 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a476d1664de7456ab21ddbf2e12b4f0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.873473 381 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.875366 381 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.875782 29160 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
W20240502 14:01:22.878270 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a476d1664de7456ab21ddbf2e12b4f0d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.878479 28881 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.878695 29292 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:22.886013 30916 raft_consensus.cc:1203] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:22.886560 32549 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:22.886849 32549 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:22.887023 32549 ts_tablet_manager.cc:1367] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.001s
I20240502 14:01:22.887496 32549 raft_consensus.cc:348] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.887605 32549 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:22.887806 32549 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:22.889921 29292 consensus_queue.cc:237] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:22.890383 32549 ts_tablet_manager.cc:1398] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:01:22.891762 29028 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.892174 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-d5ffcf19e66e4adca28d87cbe2245964 received from {username='slave'} at 127.16.217.8:51565
I20240502 14:01:22.892230 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session c59dc7abff9749abab221ce409d49f59-d5ffcf19e66e4adca28d87cbe2245964 on tablet d5ffcf19e66e4adca28d87cbe2245964 with peer c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.892769 29227 consensus_peers.cc:485] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20e960d4e55047918a1dad82fae21ecd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.892874 31907 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.894573 29160 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.894925 31907 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.901734 29159 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.902933 29425 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:22.903234 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa944cd0b82429da02bd3dd35e0cce7. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:22.903272 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa944cd0b82429da02bd3dd35e0cce7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.903321 627 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.903853 29028 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:22.904143 627 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:22.905033 31907 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.906870 29028 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.908763 28882 catalog_manager.cc:5484] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.911736 627 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.912806 29160 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:22.913797 29028 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.913813 29425 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.916494 28880 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.920697 723 ts_tablet_manager.cc:902] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.921222 723 tablet_copy_client.cc:308] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.921700 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet d5ffcf19e66e4adca28d87cbe2245964 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:49255)
I20240502 14:01:22.921787 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet d5ffcf19e66e4adca28d87cbe2245964 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:49255: session id = 12348ca540074b32912fdd6333484e0b-d5ffcf19e66e4adca28d87cbe2245964
I20240502 14:01:22.922740 29048 tablet_copy_source_session.cc:205] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.923689 723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5ffcf19e66e4adca28d87cbe2245964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.927543 723 tablet_copy_client.cc:783] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.928411 723 tablet_copy_client.cc:647] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.929988 723 tablet_copy_client.cc:523] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.935374 723 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.956802 29027 consensus_queue.cc:237] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.958417 29425 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.958844 32057 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.961256 29160 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:01:22.961728 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9e0261898a445949f3cf21d2efb599f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.962610 32218 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.962859 29292 consensus_queue.cc:237] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:22.964658 29027 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.965040 32141 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:01:22.965994 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0669ab13586a48ff9382fe1b80f47553. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.962873 804 ts_tablet_manager.cc:902] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:22.966349 29426 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.966775 804 tablet_copy_client.cc:308] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:22.967267 32057 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.967554 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 5a91fd318f0844c1961f16823a7ccc88 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:49255)
I20240502 14:01:22.967626 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 5a91fd318f0844c1961f16823a7ccc88 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:49255: session id = 12348ca540074b32912fdd6333484e0b-5a91fd318f0844c1961f16823a7ccc88
I20240502 14:01:22.968606 29048 tablet_copy_source_session.cc:205] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.968696 29425 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.969405 804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.969477 32141 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.972292 28883 catalog_manager.cc:5484] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.973821 29028 consensus_queue.cc:237] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:22.974642 804 tablet_copy_client.cc:783] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.974799 804 tablet_copy_client.cc:647] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:22.975278 29160 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
W20240502 14:01:22.976405 28961 consensus_peers.cc:485] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 372e9cc5aaed4b80b3b451efdb3e94fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:22.977547 29292 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.978572 32218 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.979832 29426 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:01:22.982573 32218 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:01:22.983323 804 tablet_copy_client.cc:523] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:22.984788 804 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:22.988008 32635 ts_tablet_manager.cc:902] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:22.988538 32635 tablet_copy_client.cc:308] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:22.993214 32141 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.994721 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet 4b1230a7230f48558da1069a5988b981 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:57785)
I20240502 14:01:22.994832 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet 4b1230a7230f48558da1069a5988b981 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:57785: session id = 22dbbf207ba0440392e71a10c2ca78ca-4b1230a7230f48558da1069a5988b981
I20240502 14:01:22.995383 29027 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.995642 29425 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:22.995775 29448 tablet_copy_source_session.cc:205] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:22.996783 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:22.998222 32218 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:22.998873 32635 tablet_copy_client.cc:783] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:22.998999 32635 tablet_copy_client.cc:647] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.002777 29292 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:23.003827 28878 catalog_manager.cc:5484] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:23.005149 28881 catalog_manager.cc:5484] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index -1 to 4, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.006295 29425 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:23.006800 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.016615 32635 tablet_copy_client.cc:523] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.018003 32635 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:23.024808 804 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.029506 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.032730 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.033716 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.035717 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.041095 31178 raft_consensus.cc:1203] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.045876 804 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:23.046175 804 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.061s user 0.026s sys 0.000s
I20240502 14:01:23.046708 804 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.050989 804 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.051182 804 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.051529 804 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:01:23.055495 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.055855 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.057284 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.063874 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-5a91fd318f0844c1961f16823a7ccc88 received from {username='slave'} at 127.16.217.6:49255
I20240502 14:01:23.063958 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 12348ca540074b32912fdd6333484e0b-5a91fd318f0844c1961f16823a7ccc88 on tablet 5a91fd318f0844c1961f16823a7ccc88 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:23.084558 723 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.084995 723 tablet_bootstrap.cc:492] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:23.085249 723 ts_tablet_manager.cc:1367] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.150s user 0.025s sys 0.000s
I20240502 14:01:23.085700 723 raft_consensus.cc:348] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.085878 723 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.086092 723 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.087067 723 ts_tablet_manager.cc:1398] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:23.088670 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-d5ffcf19e66e4adca28d87cbe2245964 received from {username='slave'} at 127.16.217.6:49255
I20240502 14:01:23.088738 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 12348ca540074b32912fdd6333484e0b-d5ffcf19e66e4adca28d87cbe2245964 on tablet d5ffcf19e66e4adca28d87cbe2245964 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:23.098902 32549 ts_tablet_manager.cc:902] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:23.099257 32549 tablet_copy_client.cc:308] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:23.099634 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 252f4087c0e5443093e856fe676839b5 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:51565)
I20240502 14:01:23.099731 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 252f4087c0e5443093e856fe676839b5 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:51565: session id = c59dc7abff9749abab221ce409d49f59-252f4087c0e5443093e856fe676839b5
I20240502 14:01:23.100590 29048 tablet_copy_source_session.cc:205] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.101364 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 252f4087c0e5443093e856fe676839b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.103246 32549 tablet_copy_client.cc:783] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.103405 32549 tablet_copy_client.cc:647] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.108166 32549 tablet_copy_client.cc:523] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.109349 32549 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:23.118721 32635 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.118976 32635 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:23.119145 32635 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.101s user 0.022s sys 0.000s
I20240502 14:01:23.119518 32635 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.119632 32635 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.121371 32635 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.122187 32635 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:01:23.124706 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-4b1230a7230f48558da1069a5988b981 received from {username='slave'} at 127.16.217.5:57785
I20240502 14:01:23.124799 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-4b1230a7230f48558da1069a5988b981 on tablet 4b1230a7230f48558da1069a5988b981 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:23.127851 31047 raft_consensus.cc:1203] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.145125 31178 raft_consensus.cc:1203] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:23.168872 32549 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.169152 32549 tablet_bootstrap.cc:492] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:23.169309 32549 ts_tablet_manager.cc:1367] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.060s user 0.019s sys 0.006s
I20240502 14:01:23.169780 32549 raft_consensus.cc:348] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:23.169903 32549 raft_consensus.cc:729] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.170045 32549 consensus_queue.cc:260] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:23.170293 32549 ts_tablet_manager.cc:1398] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:23.171579 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-252f4087c0e5443093e856fe676839b5 received from {username='slave'} at 127.16.217.8:51565
I20240502 14:01:23.171655 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session c59dc7abff9749abab221ce409d49f59-252f4087c0e5443093e856fe676839b5 on tablet 252f4087c0e5443093e856fe676839b5 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:23.175036 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.180874 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.188951 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.189656 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.217203 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.228574 29027 consensus_queue.cc:237] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
W20240502 14:01:23.231097 28962 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9e0261898a445949f3cf21d2efb599f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:23.231954 29425 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:23.231997 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9e0261898a445949f3cf21d2efb599f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:23.232350 29160 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:23.232852 32057 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.233409 32057 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.236522 32057 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:23.238898 29160 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:23.238965 29425 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:23.239153 28880 catalog_manager.cc:5484] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:23.252485 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.266891 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.274430 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.282145 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.291234 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.294174 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.305676 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.309791 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.313166 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.314035 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.315081 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.318253 723 ts_tablet_manager.cc:902] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:23.318668 723 tablet_copy_client.cc:308] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:23.319090 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:41057)
I20240502 14:01:23.319175 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:41057: session id = 12348ca540074b32912fdd6333484e0b-4aa944cd0b82429da02bd3dd35e0cce7
I20240502 14:01:23.319983 29181 tablet_copy_source_session.cc:205] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.320926 723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.322914 723 tablet_copy_client.cc:783] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.323035 723 tablet_copy_client.cc:647] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.337901 30783 raft_consensus.cc:1203] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:23.346577 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.356406 32635 ts_tablet_manager.cc:902] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:23.356823 32635 tablet_copy_client.cc:308] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:23.357239 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet db68071f838a4e209f3db69bea6e54f3 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:23.357347 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet db68071f838a4e209f3db69bea6e54f3 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-db68071f838a4e209f3db69bea6e54f3
I20240502 14:01:23.358332 29048 tablet_copy_source_session.cc:205] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.359159 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db68071f838a4e209f3db69bea6e54f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.359910 30916 raft_consensus.cc:1203] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.362262 723 tablet_copy_client.cc:523] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.363607 32635 tablet_copy_client.cc:783] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.363683 723 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:23.363734 32635 tablet_copy_client.cc:647] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.364534 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.369395 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.372014 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.379693 30783 raft_consensus.cc:1203] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:23.389112 723 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.389402 723 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:23.389562 723 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.000s
I20240502 14:01:23.389952 723 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.390065 723 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.390195 723 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.392047 723 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:01:23.393806 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-4aa944cd0b82429da02bd3dd35e0cce7 received from {username='slave'} at 127.16.217.6:41057
I20240502 14:01:23.393874 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 12348ca540074b32912fdd6333484e0b-4aa944cd0b82429da02bd3dd35e0cce7 on tablet 4aa944cd0b82429da02bd3dd35e0cce7 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:23.394169 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.414266 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.415741 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.429296 723 ts_tablet_manager.cc:902] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:23.429695 723 tablet_copy_client.cc:308] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:23.434230 32635 tablet_copy_client.cc:523] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.434288 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.435827 32635 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:23.438697 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet a476d1664de7456ab21ddbf2e12b4f0d from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:43541)
I20240502 14:01:23.438776 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet a476d1664de7456ab21ddbf2e12b4f0d from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:43541: session id = 12348ca540074b32912fdd6333484e0b-a476d1664de7456ab21ddbf2e12b4f0d
I20240502 14:01:23.439570 29448 tablet_copy_source_session.cc:205] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.440707 723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a476d1664de7456ab21ddbf2e12b4f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.442270 31178 raft_consensus.cc:1203] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.442791 723 tablet_copy_client.cc:783] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.442916 723 tablet_copy_client.cc:647] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.444193 723 tablet_copy_client.cc:523] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.445223 723 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:23.452929 30916 raft_consensus.cc:1203] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.461779 1063 ts_tablet_manager.cc:902] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:23.462275 1063 tablet_copy_client.cc:308] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:23.462648 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 0669ab13586a48ff9382fe1b80f47553 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:53421)
I20240502 14:01:23.462738 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 0669ab13586a48ff9382fe1b80f47553 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:53421: session id = 33ebf308f6f24e6e8768ef5f14ddc785-0669ab13586a48ff9382fe1b80f47553
I20240502 14:01:23.463539 29313 tablet_copy_source_session.cc:205] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.464313 32549 ts_tablet_manager.cc:902] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:23.464422 1063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0669ab13586a48ff9382fe1b80f47553. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.469982 1063 tablet_copy_client.cc:783] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.470131 1063 tablet_copy_client.cc:647] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.472029 723 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.472349 723 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:23.472520 32549 tablet_copy_client.cc:308] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:23.472538 723 ts_tablet_manager.cc:1367] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.008s
I20240502 14:01:23.472985 723 raft_consensus.cc:348] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.473039 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 20e960d4e55047918a1dad82fae21ecd from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:53621)
I20240502 14:01:23.473093 723 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.473122 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 20e960d4e55047918a1dad82fae21ecd from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:53621: session id = c59dc7abff9749abab221ce409d49f59-20e960d4e55047918a1dad82fae21ecd
I20240502 14:01:23.473214 723 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.473894 29313 tablet_copy_source_session.cc:205] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.473899 723 ts_tablet_manager.cc:1398] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:01:23.474341 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-a476d1664de7456ab21ddbf2e12b4f0d received from {username='slave'} at 127.16.217.6:43541
I20240502 14:01:23.474416 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session 12348ca540074b32912fdd6333484e0b-a476d1664de7456ab21ddbf2e12b4f0d on tablet a476d1664de7456ab21ddbf2e12b4f0d with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:23.474915 32635 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.475042 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20e960d4e55047918a1dad82fae21ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.475205 32635 tablet_bootstrap.cc:492] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:23.475374 32635 ts_tablet_manager.cc:1367] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.000s
I20240502 14:01:23.475793 32635 raft_consensus.cc:348] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.475911 32635 raft_consensus.cc:729] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.476075 32635 consensus_queue.cc:260] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.476400 32635 ts_tablet_manager.cc:1398] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 14:01:23.476866 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-db68071f838a4e209f3db69bea6e54f3 received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:23.476951 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-db68071f838a4e209f3db69bea6e54f3 on tablet db68071f838a4e209f3db69bea6e54f3 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:23.478207 32549 tablet_copy_client.cc:783] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.478343 32549 tablet_copy_client.cc:647] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.486059 32635 ts_tablet_manager.cc:902] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:23.486516 32635 tablet_copy_client.cc:308] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:23.487098 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:33903)
I20240502 14:01:23.487190 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:33903: session id = 22dbbf207ba0440392e71a10c2ca78ca-4aa944cd0b82429da02bd3dd35e0cce7
I20240502 14:01:23.487454 1083 ts_tablet_manager.cc:902] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:23.487932 1083 tablet_copy_client.cc:308] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:23.488070 29181 tablet_copy_source_session.cc:205] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.488554 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet a476d1664de7456ab21ddbf2e12b4f0d from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:51685)
I20240502 14:01:23.488643 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet a476d1664de7456ab21ddbf2e12b4f0d from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:51685: session id = 33ebf308f6f24e6e8768ef5f14ddc785-a476d1664de7456ab21ddbf2e12b4f0d
I20240502 14:01:23.489512 29448 tablet_copy_source_session.cc:205] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.490339 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.490434 1083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a476d1664de7456ab21ddbf2e12b4f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.492146 32635 tablet_copy_client.cc:783] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.492473 1083 tablet_copy_client.cc:783] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.492621 1083 tablet_copy_client.cc:647] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.493330 32635 tablet_copy_client.cc:647] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.493702 32549 tablet_copy_client.cc:523] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.494004 1083 tablet_copy_client.cc:523] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.495051 32635 tablet_copy_client.cc:523] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.495141 32549 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:23.495847 1083 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:23.496381 32635 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:23.497547 1063 tablet_copy_client.cc:523] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.498646 1063 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:23.504998 29027 consensus_queue.cc:237] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:23.507623 29425 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:01:23.507651 28961 consensus_peers.cc:485] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 372e9cc5aaed4b80b3b451efdb3e94fc. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:23.507820 28961 consensus_peers.cc:485] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 372e9cc5aaed4b80b3b451efdb3e94fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:23.508044 378 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.508633 29293 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:23.509038 31962 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.511279 31962 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.514159 29425 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.516170 29293 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.517305 28876 catalog_manager.cc:5484] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.524191 1083 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.524585 1083 tablet_bootstrap.cc:492] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:23.524811 1083 ts_tablet_manager.cc:1367] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.004s
I20240502 14:01:23.527319 1083 raft_consensus.cc:348] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.527500 1083 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.527760 1083 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:23.528798 1083 ts_tablet_manager.cc:1398] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:23.529250 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-a476d1664de7456ab21ddbf2e12b4f0d received from {username='slave'} at 127.16.217.7:51685
I20240502 14:01:23.529321 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-a476d1664de7456ab21ddbf2e12b4f0d on tablet a476d1664de7456ab21ddbf2e12b4f0d with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:23.541262 32635 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.541533 32635 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:23.541697 32635 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.001s
I20240502 14:01:23.542187 32635 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.542292 32635 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.542402 32635 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:23.542727 32635 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:23.544978 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-4aa944cd0b82429da02bd3dd35e0cce7 received from {username='slave'} at 127.16.217.5:33903
I20240502 14:01:23.545055 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-4aa944cd0b82429da02bd3dd35e0cce7 on tablet 4aa944cd0b82429da02bd3dd35e0cce7 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:23.551122 1063 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.557296 1063 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:23.557482 1063 ts_tablet_manager.cc:1367] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.059s user 0.025s sys 0.000s
I20240502 14:01:23.557879 1063 raft_consensus.cc:348] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:23.558007 1063 raft_consensus.cc:729] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.558141 1063 consensus_queue.cc:260] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:23.559919 1063 ts_tablet_manager.cc:1398] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:01:23.562140 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-0669ab13586a48ff9382fe1b80f47553 received from {username='slave'} at 127.16.217.7:53421
I20240502 14:01:23.562227 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-0669ab13586a48ff9382fe1b80f47553 on tablet 0669ab13586a48ff9382fe1b80f47553 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:23.575770 32549 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.576072 32549 tablet_bootstrap.cc:492] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:23.576377 32549 ts_tablet_manager.cc:1367] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.081s user 0.026s sys 0.000s
I20240502 14:01:23.576839 32549 raft_consensus.cc:348] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:23.576958 32549 raft_consensus.cc:729] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:23.577544 32549 consensus_queue.cc:260] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:23.577843 32549 ts_tablet_manager.cc:1398] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:23.579325 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-20e960d4e55047918a1dad82fae21ecd received from {username='slave'} at 127.16.217.8:53621
I20240502 14:01:23.579403 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session c59dc7abff9749abab221ce409d49f59-20e960d4e55047918a1dad82fae21ecd on tablet 20e960d4e55047918a1dad82fae21ecd with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:23.601954 31178 raft_consensus.cc:1203] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:23.641724 1063 ts_tablet_manager.cc:902] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:23.642105 1063 tablet_copy_client.cc:308] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:23.642563 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet d9e0261898a445949f3cf21d2efb599f from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:54097)
I20240502 14:01:23.642680 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet d9e0261898a445949f3cf21d2efb599f from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:54097: session id = 33ebf308f6f24e6e8768ef5f14ddc785-d9e0261898a445949f3cf21d2efb599f
I20240502 14:01:23.643587 29048 tablet_copy_source_session.cc:205] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.645576 1063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0261898a445949f3cf21d2efb599f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.647886 1063 tablet_copy_client.cc:783] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.648046 1063 tablet_copy_client.cc:647] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.651641 1063 tablet_copy_client.cc:523] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.653247 1063 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:23.660533 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.661278 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.662000 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.662333 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.670243 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.678875 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.690310 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.739615 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.749389 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.753942 32549 ts_tablet_manager.cc:902] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:23.754334 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.754386 32549 tablet_copy_client.cc:308] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:23.754828 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet d9e0261898a445949f3cf21d2efb599f from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:51565)
I20240502 14:01:23.754920 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet d9e0261898a445949f3cf21d2efb599f from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:51565: session id = c59dc7abff9749abab221ce409d49f59-d9e0261898a445949f3cf21d2efb599f
I20240502 14:01:23.755820 29048 tablet_copy_source_session.cc:205] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.756892 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e0261898a445949f3cf21d2efb599f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.758831 32549 tablet_copy_client.cc:783] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.758960 32549 tablet_copy_client.cc:647] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.766525 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.778780 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.793417 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.808151 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.820087 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.824447 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.828344 32549 tablet_copy_client.cc:523] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:23.828368 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.829721 32549 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:23.834034 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.836670 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.837982 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.838209 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.847566 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.848378 30916 raft_consensus.cc:1203] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.848485 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.854256 30783 raft_consensus.cc:1203] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.867565 30916 raft_consensus.cc:1203] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.883240 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.917351 29293 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:23.917531 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.918746 29425 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:23.918896 31047 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:23.919188 31907 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.919349 410 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:23.921901 31178 raft_consensus.cc:1203] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:23.940382 30783 raft_consensus.cc:1203] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:23.940387 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.940834 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.943020 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:23.956167 1063 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:23.956933 31907 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.958380 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.958846 29425 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.962041 28881 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.962500 31047 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:23.966053 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:23.969224 32635 ts_tablet_manager.cc:902] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:23.969322 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.969616 32635 tablet_copy_client.cc:308] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:23.970002 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 372e9cc5aaed4b80b3b451efdb3e94fc from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:23.970103 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 372e9cc5aaed4b80b3b451efdb3e94fc from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-372e9cc5aaed4b80b3b451efdb3e94fc
I20240502 14:01:23.971021 29048 tablet_copy_source_session.cc:205] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:23.971797 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372e9cc5aaed4b80b3b451efdb3e94fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:23.973752 32635 tablet_copy_client.cc:783] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:23.973884 32635 tablet_copy_client.cc:647] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:23.976754 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.985698 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:23.987421 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:23.989506 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:23.993602 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:23.994185 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.002044 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:24.012598 31047 raft_consensus.cc:1203] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:01:24.014794 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.029661 32635 tablet_copy_client.cc:523] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:24.031178 32635 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:24.041026 1063 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:24.041254 1063 ts_tablet_manager.cc:1367] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.388s user 0.017s sys 0.009s
I20240502 14:01:24.041644 1063 raft_consensus.cc:348] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.041761 1063 raft_consensus.cc:729] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:24.044677 1063 consensus_queue.cc:260] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.045008 1063 ts_tablet_manager.cc:1398] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240502 14:01:24.048520 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-d9e0261898a445949f3cf21d2efb599f received from {username='slave'} at 127.16.217.7:54097
I20240502 14:01:24.048595 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-d9e0261898a445949f3cf21d2efb599f on tablet d9e0261898a445949f3cf21d2efb599f with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:24.050546 1407 ts_tablet_manager.cc:902] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:24.056689 32549 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:24.056948 32549 tablet_bootstrap.cc:492] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:24.057111 32549 ts_tablet_manager.cc:1367] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.227s user 0.022s sys 0.004s
I20240502 14:01:24.057559 32549 raft_consensus.cc:348] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.057662 32549 raft_consensus.cc:729] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:24.057766 32549 consensus_queue.cc:260] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.058012 32549 ts_tablet_manager.cc:1398] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:24.059187 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-d9e0261898a445949f3cf21d2efb599f received from {username='slave'} at 127.16.217.8:51565
I20240502 14:01:24.059257 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session c59dc7abff9749abab221ce409d49f59-d9e0261898a445949f3cf21d2efb599f on tablet d9e0261898a445949f3cf21d2efb599f with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:24.060840 1407 tablet_copy_client.cc:308] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:24.060925 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.061316 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 372e9cc5aaed4b80b3b451efdb3e94fc from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:54097)
I20240502 14:01:24.061393 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 372e9cc5aaed4b80b3b451efdb3e94fc from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:54097: session id = 33ebf308f6f24e6e8768ef5f14ddc785-372e9cc5aaed4b80b3b451efdb3e94fc
I20240502 14:01:24.062198 29048 tablet_copy_source_session.cc:205] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:24.063373 1407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372e9cc5aaed4b80b3b451efdb3e94fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:24.065275 1407 tablet_copy_client.cc:783] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:24.065404 1407 tablet_copy_client.cc:647] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:24.068341 32635 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:24.068491 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.068737 32635 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:24.068892 32635 ts_tablet_manager.cc:1367] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.004s
I20240502 14:01:24.069345 32635 raft_consensus.cc:348] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:24.069449 32635 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:24.069571 32635 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:24.070094 1392 tablet_replica.cc:330] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:24.070397 1392 raft_consensus.cc:2229] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.070581 1392 raft_consensus.cc:2258] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.071430 32635 ts_tablet_manager.cc:1398] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:01:24.073158 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-372e9cc5aaed4b80b3b451efdb3e94fc received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:24.073228 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-372e9cc5aaed4b80b3b451efdb3e94fc on tablet 372e9cc5aaed4b80b3b451efdb3e94fc with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:24.073165 1392 ts_tablet_manager.cc:1875] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.076566 1392 ts_tablet_manager.cc:1888] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:24.076850 1392 log.cc:1196] T 0176fb1a25a94c1c9d8cf0dc63577e91 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/0176fb1a25a94c1c9d8cf0dc63577e91
I20240502 14:01:24.077477 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 0176fb1a25a94c1c9d8cf0dc63577e91 (table rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]) successfully deleted
I20240502 14:01:24.088513 1407 tablet_copy_client.cc:523] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:24.089989 1407 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:24.093374 31047 raft_consensus.cc:1203] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:24.119096 29160 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:24.119283 31047 raft_consensus.cc:1203] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:24.120353 29027 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:24.120771 31178 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:24.120954 29425 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:24.121685 626 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:24.121716 30916 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.121807 29425 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.122404 626 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:24.122867 32057 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.123036 378 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.123852 31047 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:24.126385 626 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:24.128221 32057 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.131475 30916 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.132573 28880 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.133038 626 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:24.136178 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet c88f85f143d845a1b7066244c606f8ba with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:24.136757 1467 tablet_replica.cc:330] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:24.136830 1407 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:24.137127 1407 tablet_bootstrap.cc:492] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:24.137305 1407 ts_tablet_manager.cc:1367] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.002s
I20240502 14:01:24.137821 1407 raft_consensus.cc:348] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:24.137939 1407 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:24.138083 1407 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:24.138371 1407 ts_tablet_manager.cc:1398] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:24.138521 1467 raft_consensus.cc:2229] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.136160 29425 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:24.138859 1467 raft_consensus.cc:2258] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.141422 28881 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:24.142076 1467 ts_tablet_manager.cc:1875] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.142750 31178 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:24.143972 31047 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:24.144179 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-372e9cc5aaed4b80b3b451efdb3e94fc received from {username='slave'} at 127.16.217.7:54097
I20240502 14:01:24.144258 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-372e9cc5aaed4b80b3b451efdb3e94fc on tablet 372e9cc5aaed4b80b3b451efdb3e94fc with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:24.145751 1467 ts_tablet_manager.cc:1888] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:24.145881 1467 log.cc:1196] T c88f85f143d845a1b7066244c606f8ba P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/c88f85f143d845a1b7066244c606f8ba
I20240502 14:01:24.146708 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet c88f85f143d845a1b7066244c606f8ba (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:01:24.147022 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 86cfc7d5709f414aac098ae3c05a48a3 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:24.149605 1392 tablet_replica.cc:330] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:24.149762 1392 raft_consensus.cc:2229] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.149928 1392 raft_consensus.cc:2258] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.150427 1392 ts_tablet_manager.cc:1875] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.151760 31178 raft_consensus.cc:1203] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:24.153584 1392 ts_tablet_manager.cc:1888] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:24.153715 1392 log.cc:1196] T 86cfc7d5709f414aac098ae3c05a48a3 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/86cfc7d5709f414aac098ae3c05a48a3
I20240502 14:01:24.154353 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 86cfc7d5709f414aac098ae3c05a48a3 (table rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]) successfully deleted
I20240502 14:01:24.154911 29426 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.196179 29293 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:24.197722 30916 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.197743 29160 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.198139 410 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.198699 1391 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.202242 1391 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.206947 29160 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.207796 30916 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.207855 28883 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.213799 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 400da91cbe234c79859b23ebe1643eb0 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:24.215093 1392 tablet_replica.cc:330] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:24.215269 1392 raft_consensus.cc:2229] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.215402 1392 raft_consensus.cc:2258] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.215924 1392 ts_tablet_manager.cc:1875] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.224078 1392 ts_tablet_manager.cc:1888] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:24.224164 1392 log.cc:1196] T 400da91cbe234c79859b23ebe1643eb0 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/400da91cbe234c79859b23ebe1643eb0
I20240502 14:01:24.229991 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 400da91cbe234c79859b23ebe1643eb0 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:24.235699 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.237849 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.253007 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.254918 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.287278 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:24.298650 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.301617 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.318115 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.320662 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.322901 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.328634 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.328936 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:24.348471 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.349282 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.353185 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.354734 30782 raft_consensus.cc:1203] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:24.355412 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.356767 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.357074 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.364353 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.377084 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.384846 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.387722 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:24.415412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.428583 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.433774 30896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.449230 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.450908 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.464763 29273 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:01:24.465761 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.474169 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.477789 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.478701 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.487607 31047 raft_consensus.cc:1203] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:01:24.535038 29292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0"
dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50612
I20240502 14:01:24.535230 29292 raft_consensus.cc:595] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:24.571254 29292 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.573611 31178 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:24.575531 29160 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:24.575917 30915 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:24.578074 29160 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:24.579504 30782 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:24.579757 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.580396 31907 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:24.580731 1451 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.581113 31178 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:24.582091 1451 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:24.582211 1391 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:24.582613 1391 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:24.582872 1391 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:24.585855 1722 raft_consensus.cc:979] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: : Instructing follower 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:24.585947 1722 raft_consensus.cc:1067] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Signalling peer 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:24.587395 30916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0"
dest_uuid: "12348ca540074b32912fdd6333484e0b"
from {username='slave'} at 127.16.217.3:53389
I20240502 14:01:24.587567 30916 raft_consensus.cc:482] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:24.587487 1451 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.587743 30916 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:24.588654 31178 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.588871 30916 raft_consensus.cc:504] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:24.589170 31907 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.589924 30916 leader_election.cc:288] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:24.590376 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:24.590488 29160 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:24.591447 29160 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:24.591648 28876 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.592075 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400da91cbe234c79859b23ebe1643eb0" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:24.592183 29292 raft_consensus.cc:3032] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:24.592218 29292 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:01:24.592361 29292 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:24.592491 29292 raft_consensus.cc:3037] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:24.593219 28882 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.593400 29292 raft_consensus.cc:2454] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:24.594175 30850 leader_election.cc:302] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:24.594436 585 raft_consensus.cc:2781] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:24.594620 585 raft_consensus.cc:686] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Becoming Leader. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Running, Role: LEADER
I20240502 14:01:24.594880 585 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:24.595678 30782 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.598120 29426 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.598647 28884 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b reported cstate change: term changed from 1 to 2, leader changed from cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) to 12348ca540074b32912fdd6333484e0b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:24.600777 31181 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.604564 29160 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.607409 30915 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:24.607846 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e1b37b67e104154b6755c1762f8f6ff with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:24.608083 1392 tablet_replica.cc:330] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:24.608218 1392 raft_consensus.cc:2229] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.608392 1392 raft_consensus.cc:2258] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.608932 1392 ts_tablet_manager.cc:1875] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.611114 1392 ts_tablet_manager.cc:1888] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:24.611182 1392 log.cc:1196] T 5e1b37b67e104154b6755c1762f8f6ff P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/5e1b37b67e104154b6755c1762f8f6ff
I20240502 14:01:24.612259 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd44eeeec934413a5001673430e6500 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:24.612802 1756 tablet_replica.cc:330] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:24.613376 1756 raft_consensus.cc:2229] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:24.613703 1756 raft_consensus.cc:2258] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:24.614349 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 5e1b37b67e104154b6755c1762f8f6ff (table rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]) successfully deleted
I20240502 14:01:24.614938 1756 ts_tablet_manager.cc:1875] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:24.617586 1756 ts_tablet_manager.cc:1888] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:24.617720 1756 log.cc:1196] T cfd44eeeec934413a5001673430e6500 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/cfd44eeeec934413a5001673430e6500
I20240502 14:01:24.618430 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet cfd44eeeec934413a5001673430e6500 (table rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]) successfully deleted
I20240502 14:01:24.666323 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.673861 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:24.687495 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.688653 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.694388 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.701792 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:24.703207 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.703445 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.703500 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.703518 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.705367 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.713289 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.742935 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.783813 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.790488 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.790745 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:24.822216 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.826460 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.832741 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.832911 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.835865 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.845254 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.860179 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.865245 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.866794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.867202 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:24.870404 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.871480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.880115 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.880941 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:24.886075 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.893064 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.939738 29160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cfd44eeeec934413a5001673430e6500"
dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36950
I20240502 14:01:24.939924 29160 raft_consensus.cc:595] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:24.977486 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:24.979302 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:24.988672 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.989145 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.989807 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:24.993943 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:24.994972 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.013723 29160 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:25.014406 1733 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.017067 29292 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:25.017565 585 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.023054 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.033066 29292 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:25.034595 30915 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.034952 29426 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.035717 1391 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.036046 1722 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.043640 1391 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.046043 30915 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.046217 29426 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.051918 28882 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.055024 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet a75497cf22014c0cb9d079b448284b3d with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:25.060775 1972 tablet_replica.cc:330] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:25.060917 1972 raft_consensus.cc:2229] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.061044 1972 raft_consensus.cc:2258] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.061520 1972 ts_tablet_manager.cc:1875] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.064464 1972 ts_tablet_manager.cc:1888] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:25.064543 1972 log.cc:1196] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/a75497cf22014c0cb9d079b448284b3d
I20240502 14:01:25.065032 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet a75497cf22014c0cb9d079b448284b3d (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:01:25.120477 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.122268 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.133996 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.137028 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.151105 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.156208 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.158985 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.169680 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.170125 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.172530 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.183249 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.192620 31162 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.195180 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.198283 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.198714 1466 raft_consensus.cc:979] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: : Instructing follower c59dc7abff9749abab221ce409d49f59 to start an election
I20240502 14:01:25.198792 1466 raft_consensus.cc:1067] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Signalling peer c59dc7abff9749abab221ce409d49f59 to start an election
I20240502 14:01:25.199856 31181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cfd44eeeec934413a5001673430e6500"
dest_uuid: "c59dc7abff9749abab221ce409d49f59"
from {username='slave'} at 127.16.217.2:37895
I20240502 14:01:25.200021 31181 raft_consensus.cc:482] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:25.200105 31181 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.203011 31181 raft_consensus.cc:504] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.204284 31181 leader_election.cc:288] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 2 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:25.204561 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "c59dc7abff9749abab221ce409d49f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:25.204679 29426 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.205724 29426 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59dc7abff9749abab221ce409d49f59 in term 2.
I20240502 14:01:25.206235 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfd44eeeec934413a5001673430e6500" candidate_uuid: "c59dc7abff9749abab221ce409d49f59" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:25.206346 29160 raft_consensus.cc:3032] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:25.206379 29160 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:25.206523 29160 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.206677 29160 raft_consensus.cc:3037] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.207468 29160 raft_consensus.cc:2454] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59dc7abff9749abab221ce409d49f59 in term 2.
I20240502 14:01:25.207870 31115 leader_election.cc:302] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59dc7abff9749abab221ce409d49f59, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:25.212500 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.219091 29426 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.220209 31178 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.220595 382 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.222661 29160 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.223032 382 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.232174 382 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.235271 28884 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.237016 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.238447 29159 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.239784 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet ed713f89a4fd47d79f03711dd14c1334 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:25.245609 2075 raft_consensus.cc:2781] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:25.245796 2075 raft_consensus.cc:686] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 LEADER]: Becoming Leader. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Running, Role: LEADER
I20240502 14:01:25.246147 2075 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.252040 28884 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 reported cstate change: term changed from 1 to 2, leader changed from bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) to c59dc7abff9749abab221ce409d49f59 (127.16.217.8). New cstate: current_term: 2 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.252511 31181 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.259011 2084 tablet_replica.cc:330] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:25.259177 2084 raft_consensus.cc:2229] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.259320 2084 raft_consensus.cc:2258] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.259816 2084 ts_tablet_manager.cc:1875] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.269788 2084 ts_tablet_manager.cc:1888] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:25.269861 2084 log.cc:1196] T ed713f89a4fd47d79f03711dd14c1334 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed713f89a4fd47d79f03711dd14c1334
I20240502 14:01:25.272009 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet ed713f89a4fd47d79f03711dd14c1334 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:25.304862 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.304857 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.307925 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.321209 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:25.324303 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.333367 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.335510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.348428 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.380777 29292 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.382463 31047 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:25.382766 30915 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:25.383123 29426 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:25.383141 410 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:25.383955 31181 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:25.384415 410 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:25.384781 410 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:25.385251 31907 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:25.389286 1722 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.391268 31047 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.392784 29426 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.395809 28881 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.399498 31181 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.401913 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet ff65ce0568514e89894ec845fbda31f3 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:25.402161 1972 tablet_replica.cc:330] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:25.402055 30915 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.402312 1972 raft_consensus.cc:2229] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.402482 1972 raft_consensus.cc:2258] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.403044 1972 ts_tablet_manager.cc:1875] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.406750 1972 ts_tablet_manager.cc:1888] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:25.406864 1972 log.cc:1196] T ff65ce0568514e89894ec845fbda31f3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff65ce0568514e89894ec845fbda31f3
I20240502 14:01:25.407378 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet ff65ce0568514e89894ec845fbda31f3 (table rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]) successfully deleted
I20240502 14:01:25.415442 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:25.419631 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.423080 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.426470 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.427899 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.431950 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.431988 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.444800 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.484061 29426 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334"
dest_uuid: "edce308627b14fa6968d90ec77d51aec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40280
I20240502 14:01:25.484251 29426 raft_consensus.cc:595] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:25.517760 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.518419 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.521294 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.523670 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:25.525579 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.525930 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.561300 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.592229 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.641543 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.650262 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.653023 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.671687 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.675909 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.676333 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.681202 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:25.682420 2087 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.685554 29160 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:25.685963 2087 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.689841 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:25.690739 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:25.731453 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.747968 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.756165 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.764483 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.768324 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.768961 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.777262 2079 raft_consensus.cc:979] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: : Instructing follower bd40abfe1fbd48529f9ebbcc4d3abf86 to start an election
I20240502 14:01:25.777339 2079 raft_consensus.cc:1067] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Signalling peer bd40abfe1fbd48529f9ebbcc4d3abf86 to start an election
I20240502 14:01:25.777905 29160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334"
dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
from {username='slave'} at 127.16.217.4:43247
I20240502 14:01:25.778059 29160 raft_consensus.cc:482] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:25.778111 29160 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.779140 29160 raft_consensus.cc:504] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.779458 29160 leader_election.cc:288] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 2 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:01:25.780375 31178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c59dc7abff9749abab221ce409d49f59"
I20240502 14:01:25.780544 31178 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.781013 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.781332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.781432 31178 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 2.
I20240502 14:01:25.782763 29095 leader_election.cc:302] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, c59dc7abff9749abab221ce409d49f59; no voters:
I20240502 14:01:25.783597 2068 raft_consensus.cc:2781] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:25.783634 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.783671 2068 raft_consensus.cc:686] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 LEADER]: Becoming Leader. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:25.783780 2068 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.785490 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed713f89a4fd47d79f03711dd14c1334" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:25.785604 29426 raft_consensus.cc:3032] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:25.785637 29426 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:25.785771 29426 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:25.785913 29426 raft_consensus.cc:3037] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:25.786845 29426 raft_consensus.cc:2454] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 2.
I20240502 14:01:25.789048 28879 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: term changed from 1 to 2, leader changed from edce308627b14fa6968d90ec77d51aec (127.16.217.4) to bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2). New cstate: current_term: 2 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:25.790439 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.794312 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:25.806735 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.815749 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.824242 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.830260 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.843704 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.846057 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.862273 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:25.864405 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:25.874972 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.882650 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:25.886751 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:25.894091 29027 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:25.895782 30915 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.896649 29292 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.896981 31047 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.896353 1465 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 14:01:25.897819 378 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.898077 378 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.906929 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:25.918462 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:25.936724 1465 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.937126 29292 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.938887 31047 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.941478 28879 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.944898 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a91fd318f0844c1961f16823a7ccc88 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:25.946034 2449 tablet_replica.cc:330] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:25.946395 2449 raft_consensus.cc:2229] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.946669 2449 raft_consensus.cc:2258] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.948434 2449 ts_tablet_manager.cc:1875] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.949287 30915 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.951754 2449 ts_tablet_manager.cc:1888] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:25.952077 2449 log.cc:1196] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/5a91fd318f0844c1961f16823a7ccc88
I20240502 14:01:25.952836 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 5a91fd318f0844c1961f16823a7ccc88 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:25.961390 29027 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:25.963032 30783 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.963423 29160 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:25.964083 378 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.964427 378 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:25.966318 1465 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:25.968145 29160 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:25.968421 30783 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:25.971089 28880 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:25.975442 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b2d5d0f4044106bed079c3faa72447 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:25.975719 2449 tablet_replica.cc:330] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:25.975868 2449 raft_consensus.cc:2229] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:25.976012 2449 raft_consensus.cc:2258] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:25.977571 2449 ts_tablet_manager.cc:1875] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:25.980878 2449 ts_tablet_manager.cc:1888] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:25.980979 2449 log.cc:1196] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/81b2d5d0f4044106bed079c3faa72447
I20240502 14:01:25.981456 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 81b2d5d0f4044106bed079c3faa72447 (table rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]) successfully deleted
I20240502 14:01:25.986747 29292 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:25.988092 30915 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.988819 410 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.992997 29028 consensus_queue.cc:237] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:25.993518 410 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.994499 29292 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.994546 30783 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.994935 31047 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:25.994942 378 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.995312 378 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.995446 1465 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:25.996438 30915 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.997328 28880 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:25.998914 378 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:25.999518 30783 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.000375 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet a75497cf22014c0cb9d079b448284b3d with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:26.000593 2449 tablet_replica.cc:330] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:26.000738 2449 raft_consensus.cc:2229] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.000895 2449 raft_consensus.cc:2258] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.001267 29293 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:26.001504 31047 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.002012 28879 catalog_manager.cc:5484] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.003777 2449 ts_tablet_manager.cc:1875] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 14:01:26.004925 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a75497cf22014c0cb9d079b448284b3d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:26.005157 30915 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:26.005520 1717 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:26.005882 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet db68071f838a4e209f3db69bea6e54f3 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:26.005939 2449 ts_tablet_manager.cc:1888] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.005991 2449 log.cc:1196] T a75497cf22014c0cb9d079b448284b3d P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/a75497cf22014c0cb9d079b448284b3d
I20240502 14:01:26.006354 2449 tablet_replica.cc:330] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:26.006441 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet a75497cf22014c0cb9d079b448284b3d (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:01:26.006469 2449 raft_consensus.cc:2229] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.006618 2449 raft_consensus.cc:2258] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.007256 1717 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.007393 2449 ts_tablet_manager.cc:1875] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.007632 30915 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.009413 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a75497cf22014c0cb9d079b448284b3d with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:26.009646 28880 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.010470 2449 ts_tablet_manager.cc:1888] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.010609 2449 log.cc:1196] T db68071f838a4e209f3db69bea6e54f3 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/db68071f838a4e209f3db69bea6e54f3
I20240502 14:01:26.011286 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet db68071f838a4e209f3db69bea6e54f3 (table rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]) successfully deleted
I20240502 14:01:26.020346 29292 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.104857 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.105885 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.107858 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.109400 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.112473 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.130977 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.132685 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.141547 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.142772 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.144064 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.148481 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.169386 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.180178 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.180546 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.198441 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.208979 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.210070 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.217993 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.223224 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.225279 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.238562 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.261035 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.271716 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.276973 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.326467 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.331082 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.333963 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.336700 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.340504 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.344118 29426 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:26.344502 1466 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.346076 31181 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:26.348073 1466 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.355160 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.364429 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.370776 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.371860 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.373837 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.379401 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.388808 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.389272 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.411816 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.412030 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.437530 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.451449 29028 consensus_queue.cc:237] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:26.453238 31181 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.453613 2522 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.454349 31047 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.454764 2522 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.454825 29426 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.456014 2522 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.458853 2522 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:26.461983 29028 consensus_queue.cc:237] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:26.462711 31181 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:26.465379 30915 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:26.465781 378 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:26.465808 29426 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:26.468415 28882 catalog_manager.cc:5484] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:26.470346 31047 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:26.471417 29160 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:26.472019 1971 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:26.473246 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e0261898a445949f3cf21d2efb599f with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:26.473582 2780 tablet_replica.cc:330] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:26.473748 2780 raft_consensus.cc:2229] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.473925 2780 raft_consensus.cc:2258] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.474561 2780 ts_tablet_manager.cc:1875] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.477494 2780 ts_tablet_manager.cc:1888] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.477617 2780 log.cc:1196] T d9e0261898a445949f3cf21d2efb599f P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/d9e0261898a445949f3cf21d2efb599f
I20240502 14:01:26.478186 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet d9e0261898a445949f3cf21d2efb599f (table rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]) successfully deleted
I20240502 14:01:26.481215 378 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:26.484335 30915 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:26.484629 28883 catalog_manager.cc:5484] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:26.487501 29160 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:26.489575 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 785828dee4594ca8813b9418cc5dba2d with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:26.491340 2796 tablet_replica.cc:330] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:26.499223 2796 raft_consensus.cc:2229] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.499472 2796 raft_consensus.cc:2258] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.500046 2796 ts_tablet_manager.cc:1875] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.502091 29292 consensus_queue.cc:237] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:26.503652 29426 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.504220 30783 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.504338 1717 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.504570 31047 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.505007 410 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.505270 410 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.509269 1717 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.509562 2796 ts_tablet_manager.cc:1888] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:26.509634 2796 log.cc:1196] T 785828dee4594ca8813b9418cc5dba2d P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/785828dee4594ca8813b9418cc5dba2d
I20240502 14:01:26.512015 30783 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.512202 28879 catalog_manager.cc:5484] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.512409 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 785828dee4594ca8813b9418cc5dba2d (table rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]) successfully deleted
I20240502 14:01:26.514991 31047 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.517155 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1adebdbd104796b927963e86a15295 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:26.517402 2780 tablet_replica.cc:330] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:26.517570 2780 raft_consensus.cc:2229] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.517709 2780 raft_consensus.cc:2258] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.518268 2780 ts_tablet_manager.cc:1875] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.520431 29426 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.520980 2780 ts_tablet_manager.cc:1888] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.521098 2780 log.cc:1196] T 2b1adebdbd104796b927963e86a15295 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b1adebdbd104796b927963e86a15295
I20240502 14:01:26.521729 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 2b1adebdbd104796b927963e86a15295 (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:01:26.538472 29426 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:26.540805 31047 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.540941 31181 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.541424 2412 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.544598 2412 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.549970 2881 ts_tablet_manager.cc:902] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:01:26.550464 2881 tablet_copy_client.cc:308] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:01:26.551234 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet a75497cf22014c0cb9d079b448284b3d from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:53421)
I20240502 14:01:26.551332 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet a75497cf22014c0cb9d079b448284b3d from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:53421: session id = 33ebf308f6f24e6e8768ef5f14ddc785-a75497cf22014c0cb9d079b448284b3d
I20240502 14:01:26.552126 29313 tablet_copy_source_session.cc:205] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:26.552937 2881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:26.554836 2881 tablet_copy_client.cc:783] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:26.554972 2881 tablet_copy_client.cc:647] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:26.556267 2881 tablet_copy_client.cc:523] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:26.557412 2881 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:26.557651 29160 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.558065 2864 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.559303 2412 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.560977 31047 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.562528 31181 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.564994 28879 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 reported cstate change: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.567262 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet f05a1a0bf1314d098f2bf65df58222e2 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:26.570047 29160 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.572887 2905 tablet_replica.cc:330] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:26.573864 2905 raft_consensus.cc:2229] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.575198 2905 raft_consensus.cc:2258] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.576869 2905 ts_tablet_manager.cc:1875] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.585567 2881 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:26.585880 2881 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:26.586046 2881 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20240502 14:01:26.586516 2881 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:26.586632 2881 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:26.586766 2881 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:26.587095 2881 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:01:26.587568 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-a75497cf22014c0cb9d079b448284b3d received from {username='slave'} at 127.16.217.7:53421
I20240502 14:01:26.587659 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-a75497cf22014c0cb9d079b448284b3d on tablet a75497cf22014c0cb9d079b448284b3d with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:26.589205 2905 ts_tablet_manager.cc:1888] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.589315 2905 log.cc:1196] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/f05a1a0bf1314d098f2bf65df58222e2
I20240502 14:01:26.597051 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet f05a1a0bf1314d098f2bf65df58222e2 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:26.631822 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.636318 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.636665 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.640051 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.642102 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.653198 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.669584 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.685911 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.687273 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.691918 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.692241 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.695652 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.705083 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.713811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.736454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.764406 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.798008 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.805570 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.808609 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.819802 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.820219 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.820292 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.821696 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.828456 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.851774 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.858774 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.863114 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:26.864679 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:26.872480 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:26.873000 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:26.874399 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.876374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:26.908710 29160 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:26.911244 30915 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.911904 29425 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.912025 2744 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.914240 30783 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.915563 1466 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.915968 3077 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.929687 29028 consensus_queue.cc:237] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:26.931351 31047 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.931649 30782 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.932111 1971 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.932422 1971 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.933254 29292 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:26.933782 378 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:26.935621 378 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.936759 29292 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.937296 3077 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.938191 31047 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.938661 28883 catalog_manager.cc:5484] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.939580 30915 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.941674 28879 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.942466 30783 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.947363 29426 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:26.947860 30782 raft_consensus.cc:2932] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:26.954778 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 372e9cc5aaed4b80b3b451efdb3e94fc with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:26.955318 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 015b587864d54da8aee36c3afd968e86 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:26.955513 2796 tablet_replica.cc:330] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:26.955632 2796 raft_consensus.cc:2229] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.955763 2796 raft_consensus.cc:2258] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.957943 2796 ts_tablet_manager.cc:1875] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.960124 2796 ts_tablet_manager.cc:1888] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.960188 2796 log.cc:1196] T 015b587864d54da8aee36c3afd968e86 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/015b587864d54da8aee36c3afd968e86
I20240502 14:01:26.962047 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 015b587864d54da8aee36c3afd968e86 (table rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]) successfully deleted
I20240502 14:01:26.963832 3099 tablet_replica.cc:330] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:26.964035 3099 raft_consensus.cc:2229] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:26.964251 3099 raft_consensus.cc:2258] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:26.964989 3099 ts_tablet_manager.cc:1875] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:26.972456 31047 raft_consensus.cc:1203] T a75497cf22014c0cb9d079b448284b3d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:01:26.982050 29426 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d"
dest_uuid: "edce308627b14fa6968d90ec77d51aec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40506
I20240502 14:01:26.982187 29426 raft_consensus.cc:595] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:26.986604 3099 ts_tablet_manager.cc:1888] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:26.986775 3099 log.cc:1196] T 372e9cc5aaed4b80b3b451efdb3e94fc P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/372e9cc5aaed4b80b3b451efdb3e94fc
I20240502 14:01:26.988931 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 372e9cc5aaed4b80b3b451efdb3e94fc (table rebalance_test_table_0 [id=15e53fa6a89946a5996190ce29b908ce]) successfully deleted
I20240502 14:01:27.023651 29292 consensus_queue.cc:237] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:27.023931 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.025378 29028 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.026046 2523 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.026736 31047 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.027338 410 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.030784 410 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:27.033604 29028 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:27.035706 28876 catalog_manager.cc:5484] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:27.038460 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.040045 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 0669ab13586a48ff9382fe1b80f47553 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:27.040439 3099 tablet_replica.cc:330] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:27.040571 3099 raft_consensus.cc:2229] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.040702 3099 raft_consensus.cc:2258] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.041426 3099 ts_tablet_manager.cc:1875] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.042503 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:27.043058 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.043978 3099 ts_tablet_manager.cc:1888] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:27.044107 3099 log.cc:1196] T 0669ab13586a48ff9382fe1b80f47553 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/0669ab13586a48ff9382fe1b80f47553
I20240502 14:01:27.044667 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 0669ab13586a48ff9382fe1b80f47553 (table rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]) successfully deleted
I20240502 14:01:27.056859 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.061049 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.061532 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.063011 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.068995 31047 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:27.121223 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.124903 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.133217 30896 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.152540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.156073 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.171124 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:27.172998 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.185490 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.252007 2898 raft_consensus.cc:979] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec: : Instructing follower 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:27.252127 2898 raft_consensus.cc:1067] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Signalling peer 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:27.252822 30916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d"
dest_uuid: "12348ca540074b32912fdd6333484e0b"
from {username='slave'} at 127.16.217.4:50627
I20240502 14:01:27.252965 30916 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:27.253029 30916 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:27.254134 30916 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:27.254696 30916 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:27.256089 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:27.256217 29426 raft_consensus.cc:3032] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:27.256260 29426 raft_consensus.cc:729] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:27.256426 29426 consensus_queue.cc:260] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:27.256677 29426 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:27.257802 29426 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:27.261504 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:27.261626 29160 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:27.262641 29160 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:27.262660 29028 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:27.263067 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:27.263175 29292 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:27.263764 30916 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:27.264124 29292 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:27.264115 1465 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.264454 30849 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:27.264729 585 raft_consensus.cc:2781] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:27.264830 585 raft_consensus.cc:686] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Becoming Leader. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Running, Role: LEADER
I20240502 14:01:27.264963 585 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:27.266880 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.267975 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.268718 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.269822 28884 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b reported cstate change: term changed from 1 to 2, leader changed from edce308627b14fa6968d90ec77d51aec (127.16.217.4) to 12348ca540074b32912fdd6333484e0b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:27.270432 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.284242 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.289678 31027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.289678 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.294363 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.305328 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.312127 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.316546 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.325726 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.355295 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.370994 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.372159 1465 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:27.375227 28880 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:27.377475 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.378132 31047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "33ebf308f6f24e6e8768ef5f14ddc785"
I20240502 14:01:27.378312 31047 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:27.378517 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet c88f85f143d845a1b7066244c606f8ba with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:27.378696 3099 tablet_replica.cc:330] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:27.378808 3099 raft_consensus.cc:2229] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.378926 3099 raft_consensus.cc:2258] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.379388 31047 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:27.379503 3099 ts_tablet_manager.cc:1875] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.381453 3099 ts_tablet_manager.cc:1888] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:27.381534 3099 log.cc:1196] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/c88f85f143d845a1b7066244c606f8ba
I20240502 14:01:27.383275 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet c88f85f143d845a1b7066244c606f8ba (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:01:27.389070 29028 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
W20240502 14:01:27.390563 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c88f85f143d845a1b7066244c606f8ba. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:27.390945 30916 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:27.397579 30915 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:27.398267 1971 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:27.398635 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.403262 378 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.404969 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c88f85f143d845a1b7066244c606f8ba with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:27.406932 28880 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.410888 30915 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.422037 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.425879 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.428071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.431571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.435042 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.439322 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.469439 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.471091 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.506596 29160 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:27.508520 30915 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:27.508577 29426 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:27.508956 1466 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.509148 30782 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:27.509454 1466 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.509738 1466 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.513420 3441 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.513890 30782 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.516443 29426 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.518977 28881 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.519971 30916 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.522410 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa944cd0b82429da02bd3dd35e0cce7 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:27.551748 3447 tablet_replica.cc:330] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:27.553009 3447 raft_consensus.cc:2229] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.554744 3447 raft_consensus.cc:2258] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.556654 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.560345 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.556732 3447 ts_tablet_manager.cc:1875] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.572693 3447 ts_tablet_manager.cc:1888] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:27.575122 3447 log.cc:1196] T 4aa944cd0b82429da02bd3dd35e0cce7 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/4aa944cd0b82429da02bd3dd35e0cce7
I20240502 14:01:27.580818 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.581878 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 4aa944cd0b82429da02bd3dd35e0cce7 (table rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]) successfully deleted
I20240502 14:01:27.590561 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.594281 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.598626 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.598659 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.603590 29292 consensus_queue.cc:237] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:27.606355 29160 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.606702 2523 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.608697 31181 raft_consensus.cc:1261] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.609138 2523 consensus_queue.cc:1035] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.616204 29028 consensus_queue.cc:237] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:27.617673 31178 raft_consensus.cc:1261] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.618067 3380 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.618132 29292 raft_consensus.cc:1261] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.619779 3380 consensus_queue.cc:1035] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.629758 1971 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.630180 31178 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.632330 29292 raft_consensus.cc:2932] T dcc536794ffb453495d025f29ea152bc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.632508 28880 catalog_manager.cc:5484] T dcc536794ffb453495d025f29ea152bc P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.637547 2523 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.640247 29160 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.643177 28875 catalog_manager.cc:5484] T 20e960d4e55047918a1dad82fae21ecd P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.646097 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc536794ffb453495d025f29ea152bc with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:27.646327 3099 tablet_replica.cc:330] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:27.646466 3099 raft_consensus.cc:2229] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.646627 3099 raft_consensus.cc:2258] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.647094 3099 ts_tablet_manager.cc:1875] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.649348 3099 ts_tablet_manager.cc:1888] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:27.649413 3099 log.cc:1196] T dcc536794ffb453495d025f29ea152bc P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/dcc536794ffb453495d025f29ea152bc
I20240502 14:01:27.650077 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet dcc536794ffb453495d025f29ea152bc (table rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]) successfully deleted
I20240502 14:01:27.650705 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.651243 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet 20e960d4e55047918a1dad82fae21ecd with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:27.651460 3447 tablet_replica.cc:330] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:27.651602 3447 raft_consensus.cc:2229] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.651733 3447 raft_consensus.cc:2258] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.652312 3447 ts_tablet_manager.cc:1875] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.652801 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.653458 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.654562 3447 ts_tablet_manager.cc:1888] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:27.654628 3447 log.cc:1196] T 20e960d4e55047918a1dad82fae21ecd P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/20e960d4e55047918a1dad82fae21ecd
I20240502 14:01:27.655135 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet 20e960d4e55047918a1dad82fae21ecd (table rebalance_test_table_6 [id=c6db7e4785cf4c7880889eda909495f9]) successfully deleted
I20240502 14:01:27.657120 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.658170 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.661729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.668829 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.681560 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.689522 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.708194 31181 raft_consensus.cc:2932] T 20e960d4e55047918a1dad82fae21ecd P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:27.745729 29426 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:27.746124 3348 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.759636 29426 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:27.760890 29292 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.761610 30782 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:27.762470 3327 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.762755 3327 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:27.765842 3327 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.767892 30782 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.770105 28880 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.774719 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1230a7230f48558da1069a5988b981 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:27.775521 29292 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:27.783855 3635 tablet_replica.cc:330] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:27.784636 3635 raft_consensus.cc:2229] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:27.784876 3635 raft_consensus.cc:2258] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:27.785478 3635 ts_tablet_manager.cc:1875] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:27.785764 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:27.786893 3348 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.787982 3635 ts_tablet_manager.cc:1888] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:27.788093 3635 log.cc:1196] T 4b1230a7230f48558da1069a5988b981 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b1230a7230f48558da1069a5988b981
I20240502 14:01:27.788697 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 4b1230a7230f48558da1069a5988b981 (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:01:27.796521 29292 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:27.796536 31047 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:27.798105 585 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.798789 3348 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:27.817616 3703 ts_tablet_manager.cc:902] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:27.818270 3703 tablet_copy_client.cc:308] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:27.818907 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet c88f85f143d845a1b7066244c606f8ba from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:27.819059 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet c88f85f143d845a1b7066244c606f8ba from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-c88f85f143d845a1b7066244c606f8ba
I20240502 14:01:27.820071 29048 tablet_copy_source_session.cc:205] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:27.827656 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.832638 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.835727 3703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:27.836675 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.838439 3703 tablet_copy_client.cc:783] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:27.838575 3703 tablet_copy_client.cc:647] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:27.839689 3703 tablet_copy_client.cc:523] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:27.840940 3703 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:27.844772 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.853693 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.867218 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.870191 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.870425 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.880101 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.885031 3703 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:27.885357 3703 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:27.886354 3703 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.004s
I20240502 14:01:27.886809 3703 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:27.886935 3703 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:27.887063 3703 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:27.887382 3703 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:27.887831 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:27.887962 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-c88f85f143d845a1b7066244c606f8ba received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:27.888044 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-c88f85f143d845a1b7066244c606f8ba on tablet c88f85f143d845a1b7066244c606f8ba with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:27.897944 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:27.898587 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.924441 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:27.926148 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:27.941057 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:27.948558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.014935 29028 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:28.018043 30916 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:28.018724 31047 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:28.019618 1971 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.019817 3380 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.023663 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.025637 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.027269 1971 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.030831 31047 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.034282 28877 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P 33ebf308f6f24e6e8768ef5f14ddc785 reported cstate change: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.037185 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.037698 30916 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.043385 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.044620 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a91fd318f0844c1961f16823a7ccc88 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:28.046090 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.046850 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.053129 3827 tablet_replica.cc:330] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:28.053282 3827 raft_consensus.cc:2229] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.053581 3827 raft_consensus.cc:2258] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.056535 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.075773 3827 ts_tablet_manager.cc:1875] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.079373 3827 ts_tablet_manager.cc:1888] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:28.079456 3827 log.cc:1196] T 5a91fd318f0844c1961f16823a7ccc88 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a91fd318f0844c1961f16823a7ccc88
I20240502 14:01:28.081740 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 5a91fd318f0844c1961f16823a7ccc88 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:28.085005 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.097846 30916 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:28.098950 29160 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:28.099386 3659 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.102906 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.103659 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.108716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.115645 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.117208 29403 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.124435 29141 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.126744 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.153513 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.159381 3658 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.160071 29160 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.163015 28881 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.166601 29160 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b"
dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37230
I20240502 14:01:28.166707 29160 raft_consensus.cc:595] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:28.169325 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 400da91cbe234c79859b23ebe1643eb0 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:28.169543 3827 tablet_replica.cc:330] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:28.169685 3827 raft_consensus.cc:2229] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.169786 3827 raft_consensus.cc:2258] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.170382 3827 ts_tablet_manager.cc:1875] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.175290 3827 ts_tablet_manager.cc:1888] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:28.175366 3827 log.cc:1196] T 400da91cbe234c79859b23ebe1643eb0 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/400da91cbe234c79859b23ebe1643eb0
I20240502 14:01:28.176074 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 400da91cbe234c79859b23ebe1643eb0 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:28.179040 30916 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.181926 29160 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:28.182305 3659 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:28.183962 3658 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.184423 29160 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.185532 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 400da91cbe234c79859b23ebe1643eb0 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:28.186842 28882 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
W20240502 14:01:28.206019 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400da91cbe234c79859b23ebe1643eb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:28.234483 29028 consensus_queue.cc:237] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.237587 30782 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:28.237579 29160 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:28.238014 1465 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.238252 3380 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.262077 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.261922 1465 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.264207 30782 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.267704 28879 catalog_manager.cc:5484] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.270430 29160 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.272142 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 19784717201a464c84607d2a43e651c2 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:28.272890 3827 tablet_replica.cc:330] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:28.273042 3827 raft_consensus.cc:2229] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.273193 3827 raft_consensus.cc:2258] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.273694 3827 ts_tablet_manager.cc:1875] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.275992 3827 ts_tablet_manager.cc:1888] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:28.276065 3827 log.cc:1196] T 19784717201a464c84607d2a43e651c2 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/19784717201a464c84607d2a43e651c2
I20240502 14:01:28.278834 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.279203 3441 raft_consensus.cc:979] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: : Instructing follower edce308627b14fa6968d90ec77d51aec to start an election
I20240502 14:01:28.279273 3441 raft_consensus.cc:1067] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Signalling peer edce308627b14fa6968d90ec77d51aec to start an election
I20240502 14:01:28.279987 29426 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b"
dest_uuid: "edce308627b14fa6968d90ec77d51aec"
from {username='slave'} at 127.16.217.2:51543
I20240502 14:01:28.280145 29426 raft_consensus.cc:482] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:28.280216 29426 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.281209 29426 raft_consensus.cc:504] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:28.281658 29426 leader_election.cc:288] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 2 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:01:28.282447 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:28.282568 29292 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.283586 29292 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:01:28.283905 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c59dc7abff9749abab221ce409d49f59"
I20240502 14:01:28.284020 31181 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.284958 31181 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:01:28.286839 29359 leader_election.cc:302] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: c59dc7abff9749abab221ce409d49f59, cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:28.287015 3624 raft_consensus.cc:2781] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:28.287144 3624 raft_consensus.cc:686] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:28.287294 3624 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:28.289136 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:28.289247 29160 raft_consensus.cc:3032] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:28.289290 29160 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:28.289407 29160 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:28.289566 29160 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.290527 29160 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:01:28.291774 28881 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 1 to 2, leader changed from bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:28.293519 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.294499 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 19784717201a464c84607d2a43e651c2 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:28.317556 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.318449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.319008 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.334266 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.346944 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.347162 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.347802 30782 raft_consensus.cc:1203] T c88f85f143d845a1b7066244c606f8ba P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:01:28.362576 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.368009 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.370762 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.371239 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.382422 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.392586 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.393919 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.396098 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.399369 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.407598 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.415380 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.424896 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.430191 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.439745 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.451042 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.519188 29160 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:28.520958 31047 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:28.521741 29028 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:28.522879 30916 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:28.523025 3442 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.524466 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:28.525105 29426 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:28.525470 31046 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:28.526006 3659 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.526489 3441 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.526952 3441 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.528482 3659 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.528827 3659 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.529526 31048 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.529919 28881 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.531144 29028 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.533006 3659 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.534914 29160 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.537497 28879 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.536562 29426 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.540906 31047 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P 33ebf308f6f24e6e8768ef5f14ddc785 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:28.540546 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda4193aaaf459fb5049ff49541229a with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:28.541183 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet a476d1664de7456ab21ddbf2e12b4f0d with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:28.541360 3827 tablet_replica.cc:330] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:28.541497 3827 raft_consensus.cc:2229] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.541653 3827 raft_consensus.cc:2258] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.542196 4132 tablet_replica.cc:330] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:28.542380 4132 raft_consensus.cc:2229] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.542562 4132 raft_consensus.cc:2258] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.542590 3827 ts_tablet_manager.cc:1875] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.543494 4132 ts_tablet_manager.cc:1875] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.544588 3827 ts_tablet_manager.cc:1888] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:28.544643 3827 log.cc:1196] T a476d1664de7456ab21ddbf2e12b4f0d P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/a476d1664de7456ab21ddbf2e12b4f0d
I20240502 14:01:28.545115 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet a476d1664de7456ab21ddbf2e12b4f0d (table rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]) successfully deleted
I20240502 14:01:28.546682 4132 ts_tablet_manager.cc:1888] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:28.550132 4132 log.cc:1196] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/1dda4193aaaf459fb5049ff49541229a
I20240502 14:01:28.551373 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 1dda4193aaaf459fb5049ff49541229a (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:01:28.613174 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.619655 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.628140 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.634310 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.635447 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.649987 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.652825 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.662814 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.664448 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.667882 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.673396 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.685623 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.707134 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.723140 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.723243 4241 ts_tablet_manager.cc:902] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:01:28.723728 4241 tablet_copy_client.cc:308] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.6:38737
I20240502 14:01:28.732683 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.741729 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.759418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.763315 29141 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:01:28.767438 30762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.768199 29160 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:01:28.768608 3996 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.769412 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.795580 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.797242 29159 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "015b587864d54da8aee36c3afd968e86"
dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37370
I20240502 14:01:28.797371 29159 raft_consensus.cc:595] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:28.797921 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.798547 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.806702 3442 raft_consensus.cc:979] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86: : Instructing follower 22dbbf207ba0440392e71a10c2ca78ca to start an election
I20240502 14:01:28.806785 3442 raft_consensus.cc:1067] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Signalling peer 22dbbf207ba0440392e71a10c2ca78ca to start an election
I20240502 14:01:28.807174 30782 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "015b587864d54da8aee36c3afd968e86"
dest_uuid: "22dbbf207ba0440392e71a10c2ca78ca"
from {username='slave'} at 127.16.217.2:47449
I20240502 14:01:28.807350 30782 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:28.807441 30782 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.808562 30782 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.811556 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.811964 30936 tablet_copy_service.cc:140] P 12348ca540074b32912fdd6333484e0b: Received BeginTabletCopySession request for tablet 400da91cbe234c79859b23ebe1643eb0 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:40477)
I20240502 14:01:28.812085 30936 tablet_copy_service.cc:161] P 12348ca540074b32912fdd6333484e0b: Beginning new tablet copy session on tablet 400da91cbe234c79859b23ebe1643eb0 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:40477: session id = 22dbbf207ba0440392e71a10c2ca78ca-400da91cbe234c79859b23ebe1643eb0
I20240502 14:01:28.812340 30782 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [CANDIDATE]: Term 2 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:01:28.812628 30916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "22dbbf207ba0440392e71a10c2ca78ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "12348ca540074b32912fdd6333484e0b"
I20240502 14:01:28.812772 30916 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.813735 30916 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dbbf207ba0440392e71a10c2ca78ca in term 2.
I20240502 14:01:28.813922 30936 tablet_copy_source_session.cc:205] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:28.814688 29160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "22dbbf207ba0440392e71a10c2ca78ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:28.814805 29160 raft_consensus.cc:3032] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:28.814836 29160 raft_consensus.cc:729] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:01:28.814954 29160 consensus_queue.cc:260] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.815112 29160 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.816016 29160 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dbbf207ba0440392e71a10c2ca78ca in term 2.
I20240502 14:01:28.817044 30717 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, 22dbbf207ba0440392e71a10c2ca78ca, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters:
I20240502 14:01:28.817277 3446 raft_consensus.cc:2781] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:28.817442 3446 raft_consensus.cc:686] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 LEADER]: Becoming Leader. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Running, Role: LEADER
I20240502 14:01:28.817731 3446 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.818145 4241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400da91cbe234c79859b23ebe1643eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:28.820122 4241 tablet_copy_client.cc:783] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:28.820292 4241 tablet_copy_client.cc:647] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:28.820780 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:01:28.821054 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "22dbbf207ba0440392e71a10c2ca78ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:28.821177 29426 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:28.821558 3996 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.821992 28877 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: term changed from 1 to 2, leader changed from bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) to 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5). New cstate: current_term: 2 leader_uuid: "22dbbf207ba0440392e71a10c2ca78ca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:28.822192 29426 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dbbf207ba0440392e71a10c2ca78ca in term 2.
I20240502 14:01:28.825855 4241 tablet_copy_client.cc:523] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:28.827109 4241 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:28.847887 29292 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:01:28.849539 4314 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:28.862978 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:28.863240 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.869637 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:28.876825 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:28.878607 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:28.885085 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:28.894963 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:28.913095 4241 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:28.913448 4241 tablet_bootstrap.cc:492] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:28.913620 4241 ts_tablet_manager.cc:1367] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.087s user 0.024s sys 0.001s
I20240502 14:01:28.914062 4241 raft_consensus.cc:348] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.914170 4241 raft_consensus.cc:729] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:28.914294 4241 consensus_queue.cc:260] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:28.914997 4241 ts_tablet_manager.cc:1398] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:01:28.915758 30936 tablet_copy_service.cc:342] P 12348ca540074b32912fdd6333484e0b: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-400da91cbe234c79859b23ebe1643eb0 received from {username='slave'} at 127.16.217.5:40477
I20240502 14:01:28.915860 30936 tablet_copy_service.cc:433] P 12348ca540074b32912fdd6333484e0b: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-400da91cbe234c79859b23ebe1643eb0 on tablet 400da91cbe234c79859b23ebe1643eb0 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:28.931056 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:28.962734 29160 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:28.964030 31178 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:28.964733 31047 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:28.965561 3442 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.966200 3441 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:28.969494 3441 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.971696 31047 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.972687 31181 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.974556 28876 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P 33ebf308f6f24e6e8768ef5f14ddc785 reported cstate change: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:28.979112 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 86cfc7d5709f414aac098ae3c05a48a3 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:28.979353 4132 tablet_replica.cc:330] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:28.979499 4132 raft_consensus.cc:2229] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:28.979677 4132 raft_consensus.cc:2258] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:28.980376 4132 ts_tablet_manager.cc:1875] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:28.982746 4132 ts_tablet_manager.cc:1888] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:28.982848 4132 log.cc:1196] T 86cfc7d5709f414aac098ae3c05a48a3 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/86cfc7d5709f414aac098ae3c05a48a3
I20240502 14:01:28.983446 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 86cfc7d5709f414aac098ae3c05a48a3 (table rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]) successfully deleted
I20240502 14:01:29.015213 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.018049 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:29.024358 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.036633 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:29.037822 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.044517 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.059146 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.061998 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:29.066565 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.076026 29028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964"
dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48074
I20240502 14:01:29.076198 29028 raft_consensus.cc:595] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:01:29.077286 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.080807 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.082378 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.084021 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.104804 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.120733 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:29.140632 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.199159 30782 raft_consensus.cc:1203] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 14:01:29.206477 30916 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Refusing update from remote peer 22dbbf207ba0440392e71a10c2ca78ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.207003 4293 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.215301 29426 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 22dbbf207ba0440392e71a10c2ca78ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.215773 4293 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.229509 29160 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 22dbbf207ba0440392e71a10c2ca78ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.230217 4293 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.234378 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.239068 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:29.239475 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.243391 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:29.250257 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.259713 29007 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:01:29.267205 1971 raft_consensus.cc:979] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: : Instructing follower 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:29.267302 1971 raft_consensus.cc:1067] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Signalling peer 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:01:29.267927 30916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964"
dest_uuid: "12348ca540074b32912fdd6333484e0b"
from {username='slave'} at 127.16.217.1:54931
I20240502 14:01:29.268050 30916 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:29.268103 30916 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.268688 30896 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:29.269194 30916 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:29.269735 30916 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:01:29.269989 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.270190 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:01:29.270290 29426 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.271251 29426 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:29.273540 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:29.273664 29292 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.274379 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:29.274488 29028 raft_consensus.cc:3032] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:29.274540 29028 raft_consensus.cc:729] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:29.274660 29292 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:29.274703 29028 consensus_queue.cc:260] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:29.274967 29028 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.274984 30850 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:29.275192 4109 raft_consensus.cc:2781] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:29.275318 4109 raft_consensus.cc:686] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Becoming Leader. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Running, Role: LEADER
I20240502 14:01:29.275442 4109 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:29.275985 29028 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:29.279986 28880 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b reported cstate change: term changed from 1 to 2, leader changed from a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) to 12348ca540074b32912fdd6333484e0b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.284514 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.284777 29028 consensus_queue.cc:237] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.285908 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:29.288628 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 785828dee4594ca8813b9418cc5dba2d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:29.289662 29159 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.289678 30916 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.290119 1971 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.290390 1971 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.294369 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.301282 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.330909 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.334157 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c59dc7abff9749abab221ce409d49f59"
I20240502 14:01:29.334301 31181 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:29.335461 31181 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:01:29.340982 1971 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.343953 29160 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.346261 28878 catalog_manager.cc:5484] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.347147 30916 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.358789 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.359782 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.361084 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.364806 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.372648 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.373199 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.388165 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.405287 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:29.407285 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.426282 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.426779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.494729 29425 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:29.496783 30782 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:01:29.497385 29359 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b1230a7230f48558da1069a5988b981. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:29.497555 29160 consensus_queue.cc:237] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.497675 3996 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.497838 29292 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.498473 3996 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.500068 30783 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:29.500712 4393 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:29.501194 29028 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:29.502204 4392 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:29.502715 4392 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.503046 30783 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.505365 4703 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:29.506168 30782 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:29.507386 29028 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.508335 29292 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:29.509519 28877 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 5 to 6, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:29.512527 28882 catalog_manager.cc:5484] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.521428 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet e6787832a36648aca61dd26a363089d5 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:29.524035 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:29.530645 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.532099 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.534200 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.541553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.556044 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.568521 4741 tablet_replica.cc:330] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:29.568797 4741 raft_consensus.cc:2229] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:29.569003 4741 raft_consensus.cc:2258] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:29.569769 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.570466 4741 ts_tablet_manager.cc:1875] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:29.574599 4741 ts_tablet_manager.cc:1888] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:29.574739 4741 log.cc:1196] T e6787832a36648aca61dd26a363089d5 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6787832a36648aca61dd26a363089d5
I20240502 14:01:29.575332 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet e6787832a36648aca61dd26a363089d5 (table rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]) successfully deleted
I20240502 14:01:29.575562 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.605644 29028 consensus_queue.cc:237] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.607748 30782 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.608151 4572 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.608688 29028 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
W20240502 14:01:29.610216 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b2d5d0f4044106bed079c3faa72447. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:29.610188 4612 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.611516 30783 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.614764 28881 catalog_manager.cc:5484] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: config changed from index 5 to 6, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.616762 30782 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.617101 4572 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.617656 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 19784717201a464c84607d2a43e651c2 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:29.618526 29027 consensus_queue.cc:237] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:29.619814 30781 raft_consensus.cc:1261] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:01:29.620113 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19784717201a464c84607d2a43e651c2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:29.620654 29160 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:01:29.620932 4572 consensus_queue.cc:1035] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.621376 4613 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.622906 4572 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:29.624328 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 19784717201a464c84607d2a43e651c2 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:29.624897 4769 tablet_replica.cc:330] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:29.625037 4769 raft_consensus.cc:2229] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:29.625169 4769 raft_consensus.cc:2258] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:29.625878 4769 ts_tablet_manager.cc:1875] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:29.629067 4769 ts_tablet_manager.cc:1888] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:01:29.629048 28881 catalog_manager.cc:5484] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:29.629199 4769 log.cc:1196] T 19784717201a464c84607d2a43e651c2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/19784717201a464c84607d2a43e651c2
I20240502 14:01:29.629473 30782 raft_consensus.cc:2932] T 19784717201a464c84607d2a43e651c2 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:29.629865 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 19784717201a464c84607d2a43e651c2 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:29.631703 4612 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.634338 29160 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.636509 28879 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 5 to 6, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.638247 30782 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.671587 29160 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.673038 30916 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:29.673393 30782 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:29.673445 4392 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.674578 4392 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.677310 4392 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.679852 30916 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.680450 28878 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.685293 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa944cd0b82429da02bd3dd35e0cce7 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:29.686007 4854 tablet_replica.cc:330] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:29.686211 4854 raft_consensus.cc:2229] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:29.686471 4854 raft_consensus.cc:2258] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:29.687163 4854 ts_tablet_manager.cc:1875] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:29.689689 4854 ts_tablet_manager.cc:1888] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:29.689826 4854 log.cc:1196] T 4aa944cd0b82429da02bd3dd35e0cce7 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/4aa944cd0b82429da02bd3dd35e0cce7
I20240502 14:01:29.690658 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 4aa944cd0b82429da02bd3dd35e0cce7 (table rebalance_test_table_2 [id=a44a326932a24e619cf3633c21b3bab0]) successfully deleted
I20240502 14:01:29.692016 30782 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:29.704416 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.708067 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.712911 29028 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:29.716480 4579 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.724906 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.726444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.740904 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.748922 29425 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:29.750084 4579 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.758764 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.763239 29292 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:29.763655 4579 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.768432 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:29.770107 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:29.780527 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.782963 31181 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:29.783488 4579 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:29.794802 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:29.814996 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.815524 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.819705 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:29.825166 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:29.832612 4941 ts_tablet_manager.cc:902] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:29.833071 4941 tablet_copy_client.cc:308] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:29.833514 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 785828dee4594ca8813b9418cc5dba2d from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:51771)
I20240502 14:01:29.833612 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 785828dee4594ca8813b9418cc5dba2d from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:51771: session id = 22dbbf207ba0440392e71a10c2ca78ca-785828dee4594ca8813b9418cc5dba2d
I20240502 14:01:29.834587 29048 tablet_copy_source_session.cc:205] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:29.835876 4941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 785828dee4594ca8813b9418cc5dba2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:29.835983 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.838589 4941 tablet_copy_client.cc:783] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:29.838729 4941 tablet_copy_client.cc:647] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:29.847545 4941 tablet_copy_client.cc:523] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:29.849123 4941 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:29.852154 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.911160 4941 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:29.912032 4941 tablet_bootstrap.cc:492] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:29.912225 4941 ts_tablet_manager.cc:1367] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.063s user 0.024s sys 0.002s
I20240502 14:01:29.912645 4941 raft_consensus.cc:348] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.912753 4941 raft_consensus.cc:729] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:29.912887 4941 consensus_queue.cc:260] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.913592 4941 ts_tablet_manager.cc:1398] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:29.913983 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-785828dee4594ca8813b9418cc5dba2d received from {username='slave'} at 127.16.217.5:51771
I20240502 14:01:29.914048 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-785828dee4594ca8813b9418cc5dba2d on tablet 785828dee4594ca8813b9418cc5dba2d with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:29.932950 30916 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:29.934295 29292 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.934330 31181 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.934710 29426 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:29.934746 4935 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.935159 4935 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.935307 4933 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:29.938043 4579 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.940169 31181 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.940757 29426 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.942742 28881 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 5 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.942915 29292 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:29.950083 29007 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ffcf19e66e4adca28d87cbe2245964 with delete_type TABLET_DATA_TOMBSTONED (TS a7221b90d0aa4770a8ce5522103a96f4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40412
I20240502 14:01:29.950474 5058 tablet_replica.cc:330] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: stopping tablet replica
I20240502 14:01:29.950661 5058 raft_consensus.cc:2229] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:29.950830 5058 raft_consensus.cc:2258] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:29.951581 5058 ts_tablet_manager.cc:1875] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:29.951695 5059 ts_tablet_manager.cc:902] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:01:29.952325 5059 tablet_copy_client.cc:308] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:01:29.952859 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet 4b1230a7230f48558da1069a5988b981 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:46805)
I20240502 14:01:29.952975 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet 4b1230a7230f48558da1069a5988b981 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:46805: session id = c59dc7abff9749abab221ce409d49f59-4b1230a7230f48558da1069a5988b981
I20240502 14:01:29.953929 29448 tablet_copy_source_session.cc:205] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:29.954818 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:29.955286 5058 ts_tablet_manager.cc:1888] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:29.955395 5058 log.cc:1196] T d5ffcf19e66e4adca28d87cbe2245964 P a7221b90d0aa4770a8ce5522103a96f4: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-0/wal/wals/d5ffcf19e66e4adca28d87cbe2245964
I20240502 14:01:29.958376 5059 tablet_copy_client.cc:783] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:29.958518 5059 tablet_copy_client.cc:647] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:29.959393 28870 catalog_manager.cc:4830] TS a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): tablet d5ffcf19e66e4adca28d87cbe2245964 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:29.960788 5059 tablet_copy_client.cc:523] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:29.962311 5059 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:01:29.970602 30916 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:29.972873 30782 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:01:29.973466 4935 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:01:29.976011 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.985782 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:29.986640 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:29.989050 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.013060 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.018640 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.019935 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.020342 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.022468 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.027137 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.032421 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.037398 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.039295 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.054579 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.054844 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.068642 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.090700 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.091243 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.095594 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.096943 4935 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.097980 5059 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:30.098315 5059 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:01:30.098528 5059 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.136s user 0.022s sys 0.004s
I20240502 14:01:30.099112 5059 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:30.099273 5059 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:01:30.099436 5059 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:30.099773 5059 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:30.100006 30782 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.102658 28884 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.104525 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-4b1230a7230f48558da1069a5988b981 received from {username='slave'} at 127.16.217.8:46805
I20240502 14:01:30.104607 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session c59dc7abff9749abab221ce409d49f59-4b1230a7230f48558da1069a5988b981 on tablet 4b1230a7230f48558da1069a5988b981 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:01:30.108417 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.109656 30916 consensus_queue.cc:237] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:30.111689 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 400da91cbe234c79859b23ebe1643eb0 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57182
W20240502 14:01:30.112413 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 400da91cbe234c79859b23ebe1643eb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.112231 30783 raft_consensus.cc:1261] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:01:30.112733 4769 tablet_replica.cc:330] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:30.112885 4769 raft_consensus.cc:2229] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.112983 4579 consensus_queue.cc:1035] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:01:30.113077 4769 raft_consensus.cc:2258] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:30.113687 4769 ts_tablet_manager.cc:1875] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:30.115001 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.115836 4769 ts_tablet_manager.cc:1888] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 14:01:30.115895 4769 log.cc:1196] T 400da91cbe234c79859b23ebe1643eb0 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/400da91cbe234c79859b23ebe1643eb0
I20240502 14:01:30.116446 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 400da91cbe234c79859b23ebe1643eb0 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:30.121028 4579 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.121179 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.123723 30783 raft_consensus.cc:2932] T 400da91cbe234c79859b23ebe1643eb0 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.124034 28884 catalog_manager.cc:5484] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 9 to 10, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.126245 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 400da91cbe234c79859b23ebe1643eb0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:30.145560 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.149726 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.208705 29292 consensus_queue.cc:237] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:30.210144 30783 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:30.210218 31047 raft_consensus.cc:1261] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:30.210590 5190 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:30.210829 4743 consensus_queue.cc:1035] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:30.213141 5190 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.215597 30783 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.216161 31047 raft_consensus.cc:2932] T 2b1adebdbd104796b927963e86a15295 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.218806 28877 catalog_manager.cc:5484] T 2b1adebdbd104796b927963e86a15295 P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.221072 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1adebdbd104796b927963e86a15295 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:30.221711 5206 tablet_replica.cc:330] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:30.222324 5206 raft_consensus.cc:2229] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.222447 29426 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:30.224465 31047 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:30.224720 31181 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:30.225365 4718 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:30.226296 4718 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:30.228864 29028 consensus_queue.cc:237] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:30.230367 29160 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:30.230880 31178 raft_consensus.cc:1261] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:01:30.230872 4779 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:30.231619 4779 consensus_queue.cc:1035] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:01:30.231833 4718 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.232136 31047 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.233493 4779 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.235153 31178 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.237699 5216 ts_tablet_manager.cc:902] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:30.237834 28884 catalog_manager.cc:5484] T 252f4087c0e5443093e856fe676839b5 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.238409 5216 tablet_copy_client.cc:308] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:30.238557 28880 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P 33ebf308f6f24e6e8768ef5f14ddc785 reported cstate change: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.239142 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 81b2d5d0f4044106bed079c3faa72447 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:49255)
I20240502 14:01:30.239233 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 81b2d5d0f4044106bed079c3faa72447 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:49255: session id = 12348ca540074b32912fdd6333484e0b-81b2d5d0f4044106bed079c3faa72447
I20240502 14:01:30.240207 29048 tablet_copy_source_session.cc:205] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:30.240720 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet 252f4087c0e5443093e856fe676839b5 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:30.241094 5216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:30.241703 29160 raft_consensus.cc:2932] T 252f4087c0e5443093e856fe676839b5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.243355 5216 tablet_copy_client.cc:783] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:30.243513 5216 tablet_copy_client.cc:647] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:30.244920 5222 tablet_replica.cc:330] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:30.245087 5222 raft_consensus.cc:2229] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.245266 5222 raft_consensus.cc:2258] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:30.245344 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet f05a1a0bf1314d098f2bf65df58222e2 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:30.245505 5216 tablet_copy_client.cc:523] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:30.245512 4769 tablet_replica.cc:330] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:30.245612 4769 raft_consensus.cc:2229] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.245756 4769 raft_consensus.cc:2258] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:30.245774 5222 ts_tablet_manager.cc:1875] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:30.246196 4769 ts_tablet_manager.cc:1875] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:30.246677 5216 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:01:30.250591 4769 ts_tablet_manager.cc:1888] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:30.250658 4769 log.cc:1196] T f05a1a0bf1314d098f2bf65df58222e2 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/f05a1a0bf1314d098f2bf65df58222e2
I20240502 14:01:30.252655 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet f05a1a0bf1314d098f2bf65df58222e2 (table rebalance_test_table_4 [id=13d99761e8c04c928e4c803a94305c14]) successfully deleted
I20240502 14:01:30.253443 5222 ts_tablet_manager.cc:1888] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:01:30.253557 5222 log.cc:1196] T 252f4087c0e5443093e856fe676839b5 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/252f4087c0e5443093e856fe676839b5
I20240502 14:01:30.254989 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 252f4087c0e5443093e856fe676839b5 (table rebalance_test_table_3 [id=752701223a054996b6bce8d893df0637]) successfully deleted
I20240502 14:01:30.268575 5206 raft_consensus.cc:2258] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:30.269229 5206 ts_tablet_manager.cc:1875] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:30.271420 5206 ts_tablet_manager.cc:1888] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:30.271543 5206 log.cc:1196] T 2b1adebdbd104796b927963e86a15295 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/2b1adebdbd104796b927963e86a15295
I20240502 14:01:30.272224 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 2b1adebdbd104796b927963e86a15295 (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:01:30.273386 5221 ts_tablet_manager.cc:902] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:01:30.273873 5221 tablet_copy_client.cc:308] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:01:30.273699 31181 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:30.274346 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 19784717201a464c84607d2a43e651c2 from peer 33ebf308f6f24e6e8768ef5f14ddc785 ({username='slave'} at 127.16.217.7:54097)
I20240502 14:01:30.274454 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 19784717201a464c84607d2a43e651c2 from peer 33ebf308f6f24e6e8768ef5f14ddc785 at {username='slave'} at 127.16.217.7:54097: session id = 33ebf308f6f24e6e8768ef5f14ddc785-19784717201a464c84607d2a43e651c2
I20240502 14:01:30.275308 29048 tablet_copy_source_session.cc:205] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:30.276716 5221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19784717201a464c84607d2a43e651c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:30.279238 5216 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:30.279495 5216 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:01:30.279743 5221 tablet_copy_client.cc:783] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:30.279867 5221 tablet_copy_client.cc:647] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:30.281311 5216 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.013s
I20240502 14:01:30.281802 5216 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:30.281912 5216 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:01:30.282042 5216 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:30.282342 5216 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:01:30.283563 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-81b2d5d0f4044106bed079c3faa72447 received from {username='slave'} at 127.16.217.6:49255
I20240502 14:01:30.283632 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 12348ca540074b32912fdd6333484e0b-81b2d5d0f4044106bed079c3faa72447 on tablet 81b2d5d0f4044106bed079c3faa72447 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:01:30.286890 5221 tablet_copy_client.cc:523] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:30.288197 5221 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap starting.
I20240502 14:01:30.320755 5221 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:30.321066 5221 tablet_bootstrap.cc:492] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Bootstrap complete.
I20240502 14:01:30.321233 5221 ts_tablet_manager.cc:1367] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.000s
I20240502 14:01:30.322708 5221 raft_consensus.cc:348] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:30.322805 5221 raft_consensus.cc:729] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ebf308f6f24e6e8768ef5f14ddc785, State: Initialized, Role: FOLLOWER
I20240502 14:01:30.322932 5221 consensus_queue.cc:260] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:30.323765 30783 raft_consensus.cc:1203] T 785828dee4594ca8813b9418cc5dba2d P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:30.325770 5221 ts_tablet_manager.cc:1398] T 19784717201a464c84607d2a43e651c2 P 33ebf308f6f24e6e8768ef5f14ddc785: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:01:30.327535 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-19784717201a464c84607d2a43e651c2 received from {username='slave'} at 127.16.217.7:54097
I20240502 14:01:30.327613 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session 33ebf308f6f24e6e8768ef5f14ddc785-19784717201a464c84607d2a43e651c2 on tablet 19784717201a464c84607d2a43e651c2 with peer 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:30.381834 31181 raft_consensus.cc:1203] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:30.392100 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.398577 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.410562 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.413000 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.414932 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.428772 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.432365 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.445288 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.447934 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.451273 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.469581 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.472358 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.478215 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.478461 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.491556 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.496996 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.500452 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.507294 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.526566 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.526863 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.530714 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.531281 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.567821 31028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:01:30.571563 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.572517 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.576472 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.582726 30762 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.592978 29007 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:01:30.609889 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.620417 29426 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:30.621678 29160 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:30.622069 4702 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:30.623559 31181 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:01:30.623973 4702 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:01:30.625715 4702 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.628010 29160 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.630255 28884 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 4 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.632481 5221 ts_tablet_manager.cc:902] T 400da91cbe234c79859b23ebe1643eb0 P 33ebf308f6f24e6e8768ef5f14ddc785: Initiating tablet copy from peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
F20240502 14:01:30.632683 5221 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.queue_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.queue_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_cpu_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_cpu_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_wall_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_wall_time_us) (tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.queue_time_us, tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.queue_time_us) (tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.run_cpu_time_us, tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.run_cpu_time_us) (tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.run_wall_time_us, tablet-download-pool-0669ab13586a48ff9382fe1b80f47553.run_wall_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.queue_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.queue_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.run_cpu_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.run_cpu_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.run_wall_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.run_wall_time_us) (tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.queue_time_us, tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.queue_time_us) (tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.run_cpu_time_us, tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.run_cpu_time_us) (tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.run_wall_time_us, tablet-download-pool-1dda4193aaaf459fb5049ff49541229a.run_wall_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.queue_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.queue_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_cpu_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_cpu_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_wall_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_wall_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.queue_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.queue_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_cpu_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_cpu_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_wall_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_wall_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.queue_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.queue_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_cpu_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_cpu_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_wall_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_wall_time_us) (tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.queue_time_us, tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.queue_time_us) (tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.run_cpu_time_us, tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.run_cpu_time_us) (tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.run_wall_time_us, tablet-download-pool-86cfc7d5709f414aac098ae3c05a48a3.run_wall_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.queue_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.queue_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_cpu_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_cpu_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_wall_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_wall_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.queue_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.queue_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_cpu_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_cpu_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_wall_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_wall_time_us) (tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.queue_time_us, tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.queue_time_us) (tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.run_cpu_time_us, tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.run_cpu_time_us) (tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.run_wall_time_us, tablet-download-pool-d9e0261898a445949f3cf21d2efb599f.run_wall_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.queue_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.queue_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_cpu_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_cpu_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_wall_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_wall_time_us) (tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.queue_time_us, tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.queue_time_us) (tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.run_cpu_time_us, tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.run_cpu_time_us) (tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.run_wall_time_us, tablet-download-pool-f05a1a0bf1314d098f2bf65df58222e2.run_wall_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.queue_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.queue_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_cpu_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_cpu_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_wall_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:01:30.639546 31181 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.646747 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.647496 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.649542 29272 tablet_service.cc:1514] Processing DeleteTablet for tablet 15513e220e144883b216c4f3ac8ade0b with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:30.649747 5222 tablet_replica.cc:330] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:30.649873 5222 raft_consensus.cc:2229] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.649999 5222 raft_consensus.cc:2258] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Raft consensus is shut down!
@ 0x7f20d8099dcd google::LogMessage::Fail() at ??:0
I20240502 14:01:30.650610 5222 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f20d809db93 google::LogMessage::SendToLog() at ??:0
@ 0x7f20d80997cc google::LogMessage::Flush() at ??:0
@ 0x7f20d809af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f20d9503044 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:01:30.669389 29028 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc"
dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38632
I20240502 14:01:30.669534 29028 raft_consensus.cc:595] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Received request to transfer leadership
@ 0x7f20d94e7f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f20d94e5efc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:01:30.673285 5222 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 14:01:30.673375 5222 log.cc:1196] T 15513e220e144883b216c4f3ac8ade0b P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/15513e220e144883b216c4f3ac8ade0b
I20240502 14:01:30.674091 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 15513e220e144883b216c4f3ac8ade0b (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
@ 0x7f20dc09877a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f20dc0a1a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f20dc12276f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f20dc1364f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f20dc121226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f20dc130c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 14:01:30.697527 5213 raft_consensus.cc:979] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4: : Instructing follower 22dbbf207ba0440392e71a10c2ca78ca to start an election
I20240502 14:01:30.697643 5213 raft_consensus.cc:1067] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Signalling peer 22dbbf207ba0440392e71a10c2ca78ca to start an election
@ 0x565423c9e75c std::function<>::operator()() at ??:0
I20240502 14:01:30.698190 30783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc"
dest_uuid: "22dbbf207ba0440392e71a10c2ca78ca"
from {username='slave'} at 127.16.217.1:58883
I20240502 14:01:30.698318 30783 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:01:30.698374 30783 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.699160 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.699585 30783 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
@ 0x7f20d94eae06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 14:01:30.700261 30783 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [CANDIDATE]: Term 2 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:30.701285 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "22dbbf207ba0440392e71a10c2ca78ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:01:30.701387 29293 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.701596 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "22dbbf207ba0440392e71a10c2ca78ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:01:30.701716 29028 raft_consensus.cc:3032] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:01:30.701758 29028 raft_consensus.cc:729] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:01:30.701912 29028 consensus_queue.cc:260] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:30.702105 29028 raft_consensus.cc:3037] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:01:30.702265 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dbbf207ba0440392e71a10c2ca78ca in term 2.
@ 0x7f20d94eb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 14:01:30.703833 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dbbf207ba0440392e71a10c2ca78ca in term 2.
I20240502 14:01:30.704159 30718 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22dbbf207ba0440392e71a10c2ca78ca, a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:01:30.704461 5114 raft_consensus.cc:2781] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:01:30.704645 5114 raft_consensus.cc:686] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [term 2 LEADER]: Becoming Leader. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Running, Role: LEADER
@ 0x7f20d94ed03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 14:01:30.704813 5114 consensus_queue.cc:237] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
@ 0x565423c9e75c std::function<>::operator()() at ??:0
I20240502 14:01:30.706893 28884 catalog_manager.cc:5484] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca reported cstate change: term changed from 1 to 2, leader changed from a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) to 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5). New cstate: current_term: 2 leader_uuid: "22dbbf207ba0440392e71a10c2ca78ca" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
@ 0x7f20d94da807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f20d9d186db start_thread at ??:0
@ 0x7f20d725071f clone at ??:0
I20240502 14:01:30.740406 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.755211 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:30.755848 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.766009 29160 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:30.766425 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.767237 31181 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:30.767621 5414 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:30.783808 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:30.803416 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:30.805233 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.808982 5414 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.811564 31178 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.812139 28884 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 2 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:30.822271 29160 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:30.824009 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet ed713f89a4fd47d79f03711dd14c1334 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:30.824059 31178 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:30.824465 4393 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:01:30.824765 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed713f89a4fd47d79f03711dd14c1334. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:30.825403 5567 tablet_replica.cc:330] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:30.826123 5567 raft_consensus.cc:2229] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:30.826218 5567 raft_consensus.cc:2258] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:30.826699 5567 ts_tablet_manager.cc:1875] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:30.829182 5567 ts_tablet_manager.cc:1888] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:30.829234 5567 log.cc:1196] T ed713f89a4fd47d79f03711dd14c1334 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/ed713f89a4fd47d79f03711dd14c1334
I20240502 14:01:30.832404 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet ed713f89a4fd47d79f03711dd14c1334 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:30.847746 30916 raft_consensus.cc:1203] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 14:01:30.852674 5562 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.854955 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ed713f89a4fd47d79f03711dd14c1334 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:30.855391 31178 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.856961 28877 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:30.941730 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:30.944147 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.952081 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.955369 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:30.981348 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:30.983733 5514 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:30.700920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.7:40969 (local address 127.16.217.5:57421)
0502 14:01:30.733211 (+ 32291us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:30.733226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:30.733299 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:30.983085 (+249786us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31966,"thread_start_us":68,"threads_started":1}
W20240502 14:01:30.983928 30717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.984987 30849 connection.cc:535] client connection to 127.16.217.7:40969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.985116 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.986577 29226 connection.cc:535] client connection to 127.16.217.7:40969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.986683 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:01:30.987185 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:30.987347 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:30.988152 28961 connection.cc:535] client connection to 127.16.217.7:40969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.988257 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.988721 29094 connection.cc:535] client connection to 127.16.217.7:40969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:30.988807 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:01:30.991215 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:30.993919 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:30.995070 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:31.000007 5677 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:30.989203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.7:40969 (local address 127.16.217.2:57019)
0502 14:01:30.999538 (+ 10335us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:30.999601 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":10270,"thread_start_us":52,"threads_started":1}
I20240502 14:01:31.004995 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.016695 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:31.016885 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.016954 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.016999 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.017027 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.017058 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.021138 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:31.069206 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.069291 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.070740 30717 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:31.080233 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:31.080359 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.086066 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.086149 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.086182 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:31.116328 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:31.117997 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.133396 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.134572 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.134764 29028 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Refusing update from remote peer 22dbbf207ba0440392e71a10c2ca78ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:31.135208 5515 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:31.148205 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.150580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.150621 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.151988 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.159765 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:31.166388 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:31.186623 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.194407 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.206482 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.207777 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.214004 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.220577 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.224996 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.237457 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.244729 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:31.247808 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.252861 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.273198 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.285442 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.286259 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.287410 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.307808 29293 raft_consensus.cc:1261] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 22dbbf207ba0440392e71a10c2ca78ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:31.308264 5515 consensus_queue.cc:1035] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:01:31.324949 30717 consensus_peers.cc:485] T 372e9cc5aaed4b80b3b451efdb3e94fc P 22dbbf207ba0440392e71a10c2ca78ca -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.364945 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.379876 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.381801 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.383119 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.410596 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.416724 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.419162 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.423250 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.424098 5924 ts_tablet_manager.cc:902] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:01:31.424108 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.424686 5924 tablet_copy_client.cc:308] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:01:31.426599 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet ed713f89a4fd47d79f03711dd14c1334 from peer 22dbbf207ba0440392e71a10c2ca78ca ({username='slave'} at 127.16.217.5:33903)
I20240502 14:01:31.426685 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet ed713f89a4fd47d79f03711dd14c1334 from peer 22dbbf207ba0440392e71a10c2ca78ca at {username='slave'} at 127.16.217.5:33903: session id = 22dbbf207ba0440392e71a10c2ca78ca-ed713f89a4fd47d79f03711dd14c1334
I20240502 14:01:31.427541 29181 tablet_copy_source_session.cc:205] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:01:31.428700 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.436631 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.437317 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.440222 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.444547 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.465916 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed713f89a4fd47d79f03711dd14c1334. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:01:31.468264 5924 tablet_copy_client.cc:783] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 0 data blocks...
I20240502 14:01:31.468451 5924 tablet_copy_client.cc:647] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Starting download of 1 WAL segments...
I20240502 14:01:31.481597 5924 tablet_copy_client.cc:523] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:01:31.481740 29426 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:31.482806 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:31.483134 5924 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap starting.
I20240502 14:01:31.483855 5227 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:31.486161 5227 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.489058 28879 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.489046 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.510071 29426 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:31.510522 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 15513e220e144883b216c4f3ac8ade0b with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:31.512300 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 14:01:31.512738 29358 consensus_peers.cc:485] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15513e220e144883b216c4f3ac8ade0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.512980 5227 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:31.515990 5227 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:31.517274 5924 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:01:31.517555 5924 tablet_bootstrap.cc:492] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Bootstrap complete.
I20240502 14:01:31.517712 5924 ts_tablet_manager.cc:1367] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.000s
I20240502 14:01:31.518165 5924 raft_consensus.cc:348] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:31.518260 5924 raft_consensus.cc:729] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbbf207ba0440392e71a10c2ca78ca, State: Initialized, Role: FOLLOWER
I20240502 14:01:31.518378 5924 consensus_queue.cc:260] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:31.518549 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 15513e220e144883b216c4f3ac8ade0b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:31.518641 5924 ts_tablet_manager.cc:1398] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:01:31.519078 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-ed713f89a4fd47d79f03711dd14c1334 received from {username='slave'} at 127.16.217.5:33903
I20240502 14:01:31.519153 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session 22dbbf207ba0440392e71a10c2ca78ca-ed713f89a4fd47d79f03711dd14c1334 on tablet ed713f89a4fd47d79f03711dd14c1334 with peer 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:31.519229 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:31.519930 28881 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) added. New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:31.522233 5983 tablet_replica.cc:330] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:31.522353 5983 raft_consensus.cc:2229] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:31.522452 5983 raft_consensus.cc:2258] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:31.524693 5983 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:31.530145 5983 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:31.530234 5983 log.cc:1196] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/15513e220e144883b216c4f3ac8ade0b
I20240502 14:01:31.531090 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 15513e220e144883b216c4f3ac8ade0b (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:01:31.543267 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.552372 31178 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:31.553088 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.554031 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:01:31.554478 4940 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:31.555583 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.572860 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.580041 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.598811 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.625397 4940 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.628407 29426 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.628527 28879 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 5 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:31.633030 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd44eeeec934413a5001673430e6500 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:31.633311 5983 tablet_replica.cc:330] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:31.633481 5983 raft_consensus.cc:2229] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:31.633590 5983 raft_consensus.cc:2258] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:31.634224 5983 ts_tablet_manager.cc:1875] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:31.638988 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.641914 5983 ts_tablet_manager.cc:1888] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:01:31.641997 5983 log.cc:1196] T cfd44eeeec934413a5001673430e6500 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/cfd44eeeec934413a5001673430e6500
I20240502 14:01:31.642556 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet cfd44eeeec934413a5001673430e6500 (table rebalance_test_table_1 [id=79af070d92f044a5b8d88e2cf5578abd]) successfully deleted
I20240502 14:01:31.647238 29028 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
W20240502 14:01:31.650266 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:31.651093 30783 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:31.652287 30916 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:31.652662 5213 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:31.654032 5213 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:31.657055 4779 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:31.658167 30916 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:31.658900 30783 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 22dbbf207ba0440392e71a10c2ca78ca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:31.660380 31178 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:31.661506 28875 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:31.664680 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:31.666406 4940 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:31.668624 4940 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:31.670490 29426 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
I20240502 14:01:31.672302 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cfd44eeeec934413a5001673430e6500 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:01:31.673597 28878 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) added. New cstate: current_term: 2 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } } }
W20240502 14:01:31.675065 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:31.677445 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.678200 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b2d5d0f4044106bed079c3faa72447 with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57182
I20240502 14:01:31.681478 5983 tablet_replica.cc:330] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:01:31.681644 5983 raft_consensus.cc:2229] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:31.681821 5983 raft_consensus.cc:2258] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:31.682328 5983 ts_tablet_manager.cc:1875] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:31.683661 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.684803 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.686323 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.691757 5983 ts_tablet_manager.cc:1888] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:31.691881 5983 log.cc:1196] T 81b2d5d0f4044106bed079c3faa72447 P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/81b2d5d0f4044106bed079c3faa72447
I20240502 14:01:31.692543 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 81b2d5d0f4044106bed079c3faa72447 (table rebalance_test_table_7 [id=3830162daeef4e3c9729520f72d4b634]) successfully deleted
W20240502 14:01:31.694267 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:31.695189 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.696687 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.706357 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.824366 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.826319 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.839227 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:31.846983 30783 raft_consensus.cc:1203] T ed713f89a4fd47d79f03711dd14c1334 P 22dbbf207ba0440392e71a10c2ca78ca [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 14:01:31.847116 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.867578 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.871500 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.872494 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.872911 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:31.881263 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:31.909024 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.909382 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.913740 5924 ts_tablet_manager.cc:902] T 15513e220e144883b216c4f3ac8ade0b P 22dbbf207ba0440392e71a10c2ca78ca: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
F20240502 14:01:31.913936 5924 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.queue_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.queue_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_cpu_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_cpu_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_wall_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_wall_time_us) (tablet-download-pool-15513e220e144883b216c4f3ac8ade0b.queue_time_us, tablet-download-pool-15513e220e144883b216c4f3ac8ade0b.queue_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.queue_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.queue_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.run_cpu_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.run_cpu_time_us) (tablet-download-pool-19784717201a464c84607d2a43e651c2.run_wall_time_us, tablet-download-pool-19784717201a464c84607d2a43e651c2.run_wall_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.queue_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.queue_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_cpu_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_cpu_time_us) (tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_wall_time_us, tablet-download-pool-2b1adebdbd104796b927963e86a15295.run_wall_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.queue_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.queue_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_cpu_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_cpu_time_us) (tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_wall_time_us, tablet-download-pool-372e9cc5aaed4b80b3b451efdb3e94fc.run_wall_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_cpu_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_cpu_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_wall_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_wall_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.queue_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.queue_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_cpu_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_cpu_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_wall_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_wall_time_us) (tablet-download-pool-4b1230a7230f48558da1069a5988b981.queue_time_us, tablet-download-pool-4b1230a7230f48558da1069a5988b981.queue_time_us) (tablet-download-pool-4b1230a7230f48558da1069a5988b981.run_cpu_time_us, tablet-download-pool-4b1230a7230f48558da1069a5988b981.run_cpu_time_us) (tablet-download-pool-4b1230a7230f48558da1069a5988b981.run_wall_time_us, tablet-download-pool-4b1230a7230f48558da1069a5988b981.run_wall_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.queue_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.queue_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_cpu_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_cpu_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_wall_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_wall_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.queue_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.queue_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_cpu_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_cpu_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_wall_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_wall_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.queue_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.queue_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_cpu_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_cpu_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_wall_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_wall_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.queue_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.queue_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_cpu_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_cpu_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_wall_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_wall_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.queue_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.queue_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_cpu_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_cpu_time_us) (tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_wall_time_us, tablet-download-pool-db68071f838a4e209f3db69bea6e54f3.run_wall_time_us) (tablet-download-pool-e6787832a36648aca61dd26a363089d5.queue_time_us, tablet-download-pool-e6787832a36648aca61dd26a363089d5.queue_time_us) (tablet-download-pool-e6787832a36648aca61dd26a363089d5.run_cpu_time_us, tablet-download-pool-e6787832a36648aca61dd26a363089d5.run_cpu_time_us) (tablet-download-pool-e6787832a36648aca61dd26a363089d5.run_wall_time_us, tablet-download-pool-e6787832a36648aca61dd26a363089d5.run_wall_time_us) (tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.queue_time_us, tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.queue_time_us) (tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.run_cpu_time_us, tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.run_cpu_time_us) (tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.run_wall_time_us, tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:01:31.914801 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.916334 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f617edf5dcd google::LogMessage::Fail() at ??:0
@ 0x7f617edf9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f617edf57cc google::LogMessage::Flush() at ??:0
@ 0x7f617edf6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:01:31.946794 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f618025f044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6180243f98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:01:31.976464 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.977933 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:31.979068 30762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:01:31.983438 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f6180241efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f6182df477a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:01:31.993230 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:31.994135 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f6182dfda45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:01:32.001495 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.002699 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
@ 0x7f6182e7e76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 14:01:32.002897 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
@ 0x7f6182e924f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:01:32.019342 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f6182e7d226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f6182e8cc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555f3593775c std::function<>::operator()() at ??:0
@ 0x7f6180246e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61802476ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f618024903b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x555f3593775c std::function<>::operator()() at ??:0
@ 0x7f6180236807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6180a746db start_thread at ??:0
@ 0x7f617dfac71f clone at ??:0
I20240502 14:01:32.057005 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.074255 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.076869 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.082504 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.084942 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.106421 29426 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:32.107951 31178 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:01:32.108356 5988 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:01:32.113310 6335 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:32.120445 31178 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:32.123325 28880 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:01:32.126389 29272 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1230a7230f48558da1069a5988b981 with delete_type TABLET_DATA_TOMBSTONED (TS cd692a8958c241fca3aec5cca3a0ded5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35958
I20240502 14:01:32.126751 6344 tablet_replica.cc:330] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: stopping tablet replica
I20240502 14:01:32.126876 6344 raft_consensus.cc:2229] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:32.127054 6344 raft_consensus.cc:2258] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:32.127552 6344 ts_tablet_manager.cc:1875] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:32.130448 6344 ts_tablet_manager.cc:1888] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:01:32.130545 6344 log.cc:1196] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b1230a7230f48558da1069a5988b981
I20240502 14:01:32.131323 28870 catalog_manager.cc:4830] TS cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): tablet 4b1230a7230f48558da1069a5988b981 (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:01:32.152127 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.156316 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.158751 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.159013 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:32.161561 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:32.164098 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.261428 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.271371 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.273970 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.281180 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.284821 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.288013 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:32.321607 30849 connection.cc:535] client connection to 127.16.217.5:45769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:32.321678 29358 connection.cc:535] client connection to 127.16.217.5:45769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:32.322523 28961 connection.cc:535] client connection to 127.16.217.5:45769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:32.329946 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.330014 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.330206 29226 connection.cc:535] client connection to 127.16.217.5:45769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:01:32.330698 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.331267 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.331326 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:32.331627 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.353510 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.364420 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.364771 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.371212 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:32.377503 30916 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:01:32.379068 31178 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:32.379426 29293 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:01:32.380065 4935 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:32.380515 4935 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:32.386433 4935 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:32.389463 31178 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:32.389577 28879 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 7 to 8, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:32.392565 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ffcf19e66e4adca28d87cbe2245964 with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:57418
I20240502 14:01:32.413182 6505 tablet_replica.cc:330] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:01:32.413344 6505 raft_consensus.cc:2229] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:01:32.413564 6505 raft_consensus.cc:2258] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:01:32.414093 6505 ts_tablet_manager.cc:1875] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:01:32.417989 6505 ts_tablet_manager.cc:1888] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 14:01:32.418145 6505 log.cc:1196] T d5ffcf19e66e4adca28d87cbe2245964 P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5ffcf19e66e4adca28d87cbe2245964
I20240502 14:01:32.418120 29293 raft_consensus.cc:2932] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:01:32.418720 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet d5ffcf19e66e4adca28d87cbe2245964 (table rebalance_test_table_8 [id=103008b3b2e64508ae956ee1a3e79272]) successfully deleted
I20240502 14:01:32.429327 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.435613 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.445137 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.453671 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.477288 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.477553 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:32.491704 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:32.491740 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:32.495223 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.503257 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.509150 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.512023 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.525430 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:32.525470 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:32.528525 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:32.532763 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:32.534716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.536049 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.536295 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.549926 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.565052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.565131 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.577494 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:32.583837 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:01:32.588707 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.588884 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.610270 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.621037 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.637195 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:32.656085 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:32.689615 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.695878 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.698004 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.700820 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.712607 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.729897 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.816540 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.818671 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.834196 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.837570 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.838172 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.847467 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.858300 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.860968 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.862959 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.885630 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:32.886162 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.897325 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.965907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:32.966624 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:32.972165 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.972738 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:32.979717 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:32.981029 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:32.983925 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:32.997128 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.005678 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.007140 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.012563 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.013156 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:33.024287 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.025851 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:33.034312 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.075016 6470 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.075135 6470 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:33.075574 6470 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:33.075847 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:33.076153 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:33.076654 30849 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 12348ca540074b32912fdd6333484e0b; no voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec
I20240502 14:01:33.076800 6470 raft_consensus.cc:2726] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:33.077116 30849 leader_election.cc:334] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.102802 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.106125 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.124317 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.124406 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.127804 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.138140 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:33.183408 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.205551 5562 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.205662 5562 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:33.206228 5562 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:33.207392 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
I20240502 14:01:33.207736 30916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "12348ca540074b32912fdd6333484e0b" is_pre_election: true
I20240502 14:01:33.207895 30916 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd40abfe1fbd48529f9ebbcc4d3abf86 in term 2.
W20240502 14:01:33.212057 29094 leader_election.cc:334] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.212119 29094 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, bd40abfe1fbd48529f9ebbcc4d3abf86; no voters: 22dbbf207ba0440392e71a10c2ca78ca, edce308627b14fa6968d90ec77d51aec
I20240502 14:01:33.212934 5562 raft_consensus.cc:2726] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:33.217077 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.223287 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.229351 6335 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.229466 6335 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:33.229959 6335 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:33.230412 30916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "12348ca540074b32912fdd6333484e0b" is_pre_election: true
I20240502 14:01:33.230538 30916 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:01:33.231302 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:01:33.231426 29159 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:01:33.231966 29358 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:01:33.232347 6335 raft_consensus.cc:2781] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:01:33.232396 6335 raft_consensus.cc:482] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting leader election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.232430 6335 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:33.233676 6335 raft_consensus.cc:504] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:33.234014 6335 leader_election.cc:288] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769)
I20240502 14:01:33.234001 6609 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.234092 6609 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:33.234517 30916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "12348ca540074b32912fdd6333484e0b"
I20240502 14:01:33.234550 6609 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:33.234629 30916 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:01:33.235750 30916 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 3.
W20240502 14:01:33.236307 29358 leader_election.cc:334] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.236852 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:33.236914 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "015b587864d54da8aee36c3afd968e86" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:01:33.237053 29159 raft_consensus.cc:3037] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:01:33.237308 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.237370 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, a7221b90d0aa4770a8ce5522103a96f4
W20240502 14:01:33.237468 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:33.237633 6609 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:33.238067 29159 raft_consensus.cc:2454] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 3.
W20240502 14:01:33.239068 29358 leader_election.cc:334] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.239203 29358 leader_election.cc:302] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters: 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:33.239404 6335 raft_consensus.cc:2781] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:01:33.239565 6335 raft_consensus.cc:686] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 3 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:01:33.239706 6335 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:01:33.243971 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.250771 28879 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 2 to 3, leader changed from 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 3 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:01:33.251169 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.252681 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.254829 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.256201 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.258724 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.260349 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.261103 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.261703 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.262414 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.268172 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:33.271656 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.271807 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.273151 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.300102 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:33.306818 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:33.311800 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.319147 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:33.335741 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.344611 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.415361 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.419661 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.424760 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.431769 7071 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:33.431870 7071 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:33.432318 7071 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:33.432742 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:33.432873 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:01:33.435232 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:33.435317 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:33.435348 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:33.435564 7071 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:33.446873 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.448099 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.452499 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:33.469749 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:33.472585 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.503993 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.515776 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.516048 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:33.520946 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.547895 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.564154 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.564882 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.569722 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.577575 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:33.593509 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.593590 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.625514 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.625658 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.642269 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.642604 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.647545 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.675379 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.683374 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.686261 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.690114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.709267 30916 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P 12348ca540074b32912fdd6333484e0b [term 3 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 14:01:33.709875 7069 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:01:33.717020 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.724960 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.730566 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.731639 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.734581 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.745000 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:33.745990 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:33.763564 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:33.779181 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:01:33.802338 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:33.805141 29159 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 14:01:33.805675 7069 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:01:33.805742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.811317 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.816466 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.823446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.825212 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.841861 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.861917 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:33.868031 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:33.870800 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:33.881165 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.883476 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.910552 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:33.982926 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:33.991744 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:33.992056 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:33.995771 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:33.998484 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:34.009580 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.011632 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.011855 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.026055 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.030349 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.030535 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.039108 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.042254 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.054808 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.066357 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.069983 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.071933 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.074071 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.088937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.120544 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:34.168555 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:34.257398 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.271643 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.286957 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.296262 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.303081 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.319372 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.329277 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.331449 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.341154 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.342835 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.343513 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.355815 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.365304 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.380049 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.380692 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.380813 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:34.383685 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.387771 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.388729 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.389060 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.414441 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.419718 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.421975 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.432911 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.450863 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:34.451655 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.454583 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.456912 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.462379 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.465180 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.465651 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.471295 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:34.484437 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:34.509135 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:34.586637 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.628763 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.630290 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.636957 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:34.644832 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:34.651903 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.658944 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.661279 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.683944 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.695689 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.695895 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.714530 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.726322 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.727427 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:34.773180 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:34.781486 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.786553 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.787379 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.787623 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.791414 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.808331 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:34.819257 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:34.834184 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:34.845815 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:01:34.851614 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.886404 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:34.889798 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.899571 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:34.908422 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:34.919123 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:34.923130 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.927320 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.944523 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:34.945333 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:34.951714 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:34.954897 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:34.962877 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:34.963863 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:34.966361 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:34.985524 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.047607 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.049512 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.052436 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.073292 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.073606 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.107967 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.114223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:35.121874 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:35.122475 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:35.124475 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.131302 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.139035 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.142289 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.147696 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.209642 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.222522 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.229104 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.241448 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:35.241726 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:35.251353 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.251619 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.252985 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.253912 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.265842 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.270787 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.274266 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.300165 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.339804 7762 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:35.339932 7762 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:35.340394 7762 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:35.341068 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:35.341204 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:01:35.342299 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:35.342411 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:35.342464 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:35.342824 7762 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:35.360987 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.369717 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.379386 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.391048 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.394032 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.400312 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.403752 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.404670 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:35.411033 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:35.415200 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.429620 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:35.437295 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:35.441893 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.444624 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.449210 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.450076 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.459414 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.464635 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.469673 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.492480 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.555429 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.562172 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.567430 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.569010 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.571952 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.573869 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:35.579123 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.589478 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.599172 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.661092 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.663674 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.664479 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.665048 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.666863 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.671311 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.681425 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.685778 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.703886 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.719033 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.723325 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:35.727963 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:35.736322 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.780745 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.789592 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.793288 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:35.800400 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.815109 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:35.824031 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.827643 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:35.841382 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.862046 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.862768 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.865447 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:35.876324 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:35.879056 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.889271 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.894196 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:35.901813 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:35.914114 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:35.915051 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:35.916538 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:35.922080 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:35.925796 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:35.934199 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:35.938426 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:35.991255 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:36.009351 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 14:01:36.018633 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 14:01:36.028672 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.035148 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.039805 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:36.043404 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:01:36.049643 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.058744 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.065445 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.068014 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.077692 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.087381 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:36.088245 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:36.092242 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:36.101397 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:36.111914 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.112612 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:36.143853 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240502 14:01:36.162045 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.163125 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.166543 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.173384 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.192875 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:36.206692 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:36.223399 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.223637 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.227771 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.231050 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.231426 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.237282 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.244657 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:36.262226 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20240502 14:01:36.263557 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:01:36.265051 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.266456 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.270092 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.272295 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.285323 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.296075 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.356257 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.359206 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.363312 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.363381 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.373875 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.375131 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:36.440099 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:36.467253 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.467478 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.470867 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.474170 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.477643 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.478247 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:36.480459 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:36.519307 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.523478 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.525050 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.544009 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.551347 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.558193 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:36.625490 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:36.657063 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.663980 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.665767 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.671960 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.672542 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.674207 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.727427 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.741549 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.750293 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.751906 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.753417 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.756958 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.776762 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.785043 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:36.787032 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:36.793630 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.801005 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.801641 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.807561 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.810194 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.827875 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.849164 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.853582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:36.855376 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.860460 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.902158 8366 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:36.902280 8366 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:36.902700 8366 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:36.903149 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:36.903271 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:01:36.904974 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:36.905092 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:36.905153 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:36.905452 8366 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:36.947837 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.951171 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:36.962071 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:36.970006 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:01:36.971158 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:36.972020 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:36.979234 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:36.979501 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:36.979537 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:36.981878 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.000164 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:37.001753 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.002447 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:37.019042 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.020308 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.049783 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:37.055471 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:37.089021 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.090404 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.106918 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.112646 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.126066 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.126654 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.126765 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.130326 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.133846 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.140062 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.140615 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.145805 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.170745 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:37.201153 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:37.251375 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.277030 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.285392 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.286877 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:37.299229 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.301807 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.302266 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.302356 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.316561 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.317684 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.326333 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.342703 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.344805 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:37.357146 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.361596 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.368149 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.373488 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.376053 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.385356 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.399121 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.400101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:37.406198 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:37.445981 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.448403 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.448683 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.452786 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.460394 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.471468 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:37.481941 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:37.505113 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:37.510857 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:37.514016 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:37.514786 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.517675 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.518765 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.523047 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:37.529000 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:37.543680 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.544296 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.634266 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.652884 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.653746 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.657958 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.664870 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.665905 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.695855 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.697696 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.704474 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.717109 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.717592 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.726898 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.730067 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.734912 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.735932 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.738283 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.740558 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.750535 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:37.760933 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:01:37.777621 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:37.845458 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.854686 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.859822 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.864446 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.874581 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.889209 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.914182 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:37.914675 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:37.916149 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.922488 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:37.927002 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:37.952033 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:37.976228 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:37.990962 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:37.995718 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:37.997140 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.000713 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.011169 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.014163 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:38.057515 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:38.076296 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.080699 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.084722 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.089373 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:38.122332 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.137444 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.138564 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.148015 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:38.155874 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:38.166512 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.167111 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.176389 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.181144 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.188138 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.199133 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.210232 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.210498 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.237043 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.238652 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.249783 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.280169 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.290388 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.293960 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.297909 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:38.301903 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.302111 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.318444 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.320899 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:38.327044 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.350962 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.362293 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:38.391690 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.402637 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.404172 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.409257 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.416110 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.416868 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.423166 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.437975 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.442306 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.457449 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.463013 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.482246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.486030 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.493562 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.501549 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.506908 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.509019 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.528052 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:38.537783 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:38.549407 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:38.552747 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.581496 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.589044 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.589310 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.594447 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.596482 9108 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:38.596585 9108 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:38.597029 9108 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:38.597484 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:38.597618 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:01:38.601342 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:38.601441 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:38.601482 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:38.604496 10017 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:38.613366 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.618940 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.672500 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.676065 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.682755 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.684959 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.706998 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.710109 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.714285 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.714807 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.716426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:38.719911 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.724705 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:38.731814 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:01:38.734257 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:38.754381 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.758055 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.794247 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.796566 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.819224 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:38.823851 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.840050 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:38.846839 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:38.849967 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.911140 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.917193 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.930564 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.934763 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.944039 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:38.945577 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.951339 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:38.966004 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:38.975973 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:38.979722 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:38.982888 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:38.983441 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.986119 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:38.992372 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:38.999341 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.010429 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.018600 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.021433 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.024876 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.040462 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.089582 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.097307 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.098040 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.098371 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:39.107520 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:39.111304 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.121914 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.168849 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.175806 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.177681 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.179277 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.187237 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.187664 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:39.246220 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:39.281668 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.286060 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.294718 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.295022 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.297331 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.319571 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.319685 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.327667 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.329380 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:39.332808 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.335662 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.335850 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.354504 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.365543 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.372043 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.373638 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.374539 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.374742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.380935 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:39.492832 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.515367 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.520247 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.530431 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.530834 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:39.536343 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.553084 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.554262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:39.565722 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.581876 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.586436 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:39.593792 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.594295 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.613652 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.630373 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:39.631628 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:39.641564 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.643550 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:39.643831 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:39.649638 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.652730 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.661938 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.663743 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.665858 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.666333 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.677356 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.678759 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.683887 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:39.696408 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:39.700465 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:39.711323 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.718854 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:39.742956 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.835055 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.837280 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.845505 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.848110 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.849015 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:39.871374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:39.878314 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.883829 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.886101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.889235 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.895630 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.896644 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.939680 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:39.940951 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:39.942384 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:39.951777 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:39.967656 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:39.969990 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:39.972788 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:39.978338 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.040062 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.044795 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:40.053251 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:40.057945 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.059466 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.059602 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.061746 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:40.073343 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:40.110965 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:01:40.121649 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:40.121997 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.125372 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.128218 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.139703 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.145129 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.158952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.167346 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.170017 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.184561 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.190943 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.195555 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.204737 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:40.238734 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:40.268575 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:40.275300 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:40.278038 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.286903 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:40.290446 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:40.302307 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.304878 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.308765 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.371223 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.377631 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.380879 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.387720 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.397644 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.422920 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.427492 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.433452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.434078 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.443192 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.444461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.444631 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.446640 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.447067 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:40.460343 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:40.460805 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.461766 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.465073 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.465695 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.629513 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:40.630460 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:40.636579 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.643026 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.650615 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.661777 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.670452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.671942 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.671981 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.674759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.676702 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.682934 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.692653 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.694633 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.715565 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.716217 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.718729 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:40.725730 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.736584 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:40.753026 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.754011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:40.771262 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:40.803700 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.810575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.812420 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.814956 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.815199 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.816385 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:40.867223 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.871686 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.873095 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.882266 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.888060 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:40.912768 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:40.914433 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:40.962162 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:40.971952 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:40.976083 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:40.978291 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:40.985675 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:40.986202 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.015681 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:41.020439 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:01:41.033149 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:41.039127 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:41.040076 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.043634 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.055050 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.055989 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.058346 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:41.065354 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:41.065495 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:41.069147 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:41.100219 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:01:41.100786 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:41.103588 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:41.107105 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:41.107834 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:01:41.112968 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.116786 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.120729 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.127313 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:41.160948 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:01:41.179728 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:01:41.198264 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.200063 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.203029 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.206837 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.216742 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.242764 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.269735 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.278888 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.280424 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.284188 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.290608 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:41.293352 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:41.323349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.329066 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.331444 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.335645 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.343070 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.349417 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.350158 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:41.382923 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:01:41.435789 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.437741 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.438040 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:41.447053 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:41.447243 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.447477 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:41.448940 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:41.463367 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.485056 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.490257 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:41.492398 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:41.494082 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.494572 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.496073 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.504653 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.555065 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.556573 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.564126 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:41.564827 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:41.578528 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.594650 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.595692 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.634575 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.640259 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.641106 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.643174 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.655680 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.667474 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.704996 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.717875 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.717932 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.729876 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.750713 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.752131 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.755219 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.763794 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.773502 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.774088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.785796 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:41.787026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.789642 7762 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:41.789737 7762 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:41.790190 7762 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:41.790520 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:41.790649 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:01:41.792201 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:41.792567 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:41.792622 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:41.793299 7762 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:01:41.824199 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:41.927495 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.929816 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.930349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.942319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.951779 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:41.956225 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:41.958459 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:41.959034 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:41.959745 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:41.973258 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:41.986770 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:41.999269 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.004711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.004947 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.007601 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.008564 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:42.027129 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:01:42.028311 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:42.037540 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:42.041267 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.047269 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.052889 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.065841 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:42.069029 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:42.075800 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.078755 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.086043 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.092604 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.095175 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.102725 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.180027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.183609 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.189524 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.194269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.207612 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.213510 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:42.221450 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:42.224349 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.236179 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.253722 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.269763 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.270488 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.290486 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.304481 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:42.305486 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.311825 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.312258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:42.314556 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.324035 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.324363 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.351456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.351716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.352629 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.360189 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.363027 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.364024 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:42.380075 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:42.380203 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.380914 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.400714 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.402081 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:42.413964 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:42.469158 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.474263 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.478747 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.486685 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.515473 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.525264 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:42.527660 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:42.533316 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.546523 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.547955 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.563944 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.564958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.565011 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.581271 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.592862 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.605885 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.606380 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.609136 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.621435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:42.632040 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:01:42.645609 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.766556 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.794729 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.809010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.813043 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.817525 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.818090 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.822058 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.824537 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.837488 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.838716 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:42.853252 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.860013 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.862011 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.874989 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.876883 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.883260 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.887351 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.890424 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.891252 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.892851 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.896742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:42.898988 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:42.907130 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:42.920467 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:42.925966 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.926431 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:42.931990 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.932698 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:42.941314 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:42.944298 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:42.956456 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:42.956946 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:42.998272 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:43.059484 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:43.067982 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.078286 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.085503 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.087987 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.090323 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.095739 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.097746 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.115062 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.117535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.136336 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.151536 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.151630 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.153677 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:43.158845 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.190646 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.193033 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.194705 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.196110 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.197258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.200862 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:43.254042 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.279703 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.307797 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.314769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.316103 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:43.318702 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.319752 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.325599 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.325848 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.335219 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.335829 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.339618 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.341286 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.348560 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.349155 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:43.382723 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:43.485231 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:43.502945 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.514200 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.518819 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.520715 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.521826 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.535909 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.553611 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:43.554068 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:01:43.565101 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:43.566978 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:43.578011 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.596725 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.599071 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.599192 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.602382 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.603794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:43.604833 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.612740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.623962 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.628291 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.630254 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.636869 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.639137 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.644778 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:43.646237 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:43.647063 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.650059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.653556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.681074 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.716936 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.720994 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.727684 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.728909 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.735662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.762691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.855348 11457 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:43.855479 11457 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:43.855929 11457 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:43.856540 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:43.856690 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:01:43.859606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:43.859853 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:43.859952 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:43.859994 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:43.860200 11457 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:43.866847 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.883999 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.898950 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:43.900444 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:43.902922 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.921319 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.925248 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.932449 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.937511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.944810 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:43.947301 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:43.954950 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:43.964578 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.964987 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:43.983757 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:43.989394 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:43.990976 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.004127 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.010852 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.032866 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.035982 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.041692 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:44.044458 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.047430 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.058941 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:44.080638 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:44.087136 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:44.119042 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:44.161407 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.166877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.176371 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.187744 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.187822 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.191699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.202540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.204470 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.214406 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.214639 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.229840 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.244122 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.259107 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.259975 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.268949 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.269423 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.284119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.300573 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:44.320120 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:44.393494 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.431367 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.435431 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.442385 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.443933 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.445024 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.449558 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.460865 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:44.465080 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:44.470770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.472474 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.487731 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.487738 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.493559 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.496768 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:44.516436 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:44.518940 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.527918 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.529026 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.530431 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.534406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.540989 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.542282 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.555321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.571117 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.571837 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.593253 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.602525 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:44.623870 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.624707 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.626168 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.626739 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.643522 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.656342 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.661432 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:44.673252 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:44.728152 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.729329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.733742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:44.741863 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:44.747059 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.754504 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.754510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.760432 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.780515 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:44.781997 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.790023 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.799372 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:44.799530 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.809511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.813338 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.891146 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.892391 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.894088 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.898353 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.904633 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:44.905535 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.918931 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:44.930956 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:44.955119 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:44.961956 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:44.964718 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:44.967981 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:44.972584 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:44.973786 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:44.982740 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.077014 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.083890 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.090338 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.091542 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.093333 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.093677 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.109552 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:45.112222 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:45.114825 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:45.123615 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:45.126698 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.149935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.153290 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.166841 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.206511 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.209508 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.212263 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.214032 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.221251 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.242192 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.285625 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.292487 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.302234 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.305166 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.309718 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.309765 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.310914 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.323695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.326709 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.328724 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:45.338407 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.343257 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:45.348506 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.356760 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:45.424897 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:01:45.433333 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:45.443917 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.445035 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.459542 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.465987 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.467355 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.469929 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.476418 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.482723 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.498868 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.504266 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.505995 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.508740 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.567320 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.571018 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:45.577195 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:45.594337 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.605615 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.605742 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.619623 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.623235 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.627038 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.639108 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.641256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.657749 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.660033 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:45.680428 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.727052 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:45.732412 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.747304 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.757436 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.758010 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.761747 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:45.790251 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.799764 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:45.802546 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.803232 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.805745 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:45.808109 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.811290 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.814859 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:45.816483 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:45.818050 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:45.833691 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:45.835276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.840669 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:45.856144 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:45.860239 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:45.874732 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:45.897311 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.968501 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:45.998242 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.014336 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.017215 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:46.021175 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:01:46.021955 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.023123 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:46.028900 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:01:46.038985 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.049580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.052855 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.069300 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.071976 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.073479 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.090628 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.091152 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.098678 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.099821 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:46.110741 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.113651 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:46.115304 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:01:46.117101 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.127117 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.148613 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.149624 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.164557 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.166231 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.166898 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.174365 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.176426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.188841 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:46.200577 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:01:46.221479 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 14:01:46.227195 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.242141 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.248664 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.251160 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.256752 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.264847 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.271776 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:46.304448 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:46.326457 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:46.330041 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.330372 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.331420 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.331956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.334751 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.335791 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.391968 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.393312 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.394052 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.394079 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.395609 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.398686 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:46.464395 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:46.488687 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:01:46.494052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.497102 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.499847 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.512737 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.517719 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.523890 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.528375 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:46.536675 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.542559 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.543910 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.544260 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.545068 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.549319 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:46.693289 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.695896 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.705107 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.714265 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.718947 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.719848 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.721400 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.729427 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:46.747324 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:46.761193 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.777673 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.787128 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:46.793534 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:46.800032 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.808038 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.821424 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.823604 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.833413 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.834563 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.846977 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.858700 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.862021 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.865487 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:46.872635 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:46.873425 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.874044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.876925 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:46.927489 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:46.929667 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.933575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:46.935217 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:46.942636 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:46.949309 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:46.952802 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.020017 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.025656 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.035032 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.051765 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.052641 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.054595 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:47.063238 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:47.087985 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.128587 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:47.142640 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.146585 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.154793 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:47.160071 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:47.179158 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.190402 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.191749 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.201084 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.206251 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:47.208523 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:47.209064 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.214998 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.216361 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:47.223940 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:47.224449 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:47.225894 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 14:01:47.226900 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.229748 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.230067 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.244575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.245494 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.255931 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:47.264816 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.267661 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.275467 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.280638 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.285023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.287447 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.290825 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:47.322614 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:47.427520 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.432389 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.441923 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.449241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.451401 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.457162 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.469208 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.473591 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.476393 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.486953 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.495303 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.496066 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:47.504734 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.514775 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.525718 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.526448 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.529011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.532560 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.542037 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:47.568522 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.679904 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.689647 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.692909 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.701937 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.709280 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:47.709653 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.713914 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.740844 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.743633 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.745465 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.747481 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.748783 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.763314 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.764478 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.780444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.785312 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.788888 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.822070 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:47.822654 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:47.822965 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:47.828704 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.843991 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:47.852735 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:47.856379 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:47.863149 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.866536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.876205 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.880146 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:47.913316 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:01:47.935956 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:47.946319 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:47.949555 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:47.950719 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:47.955806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:47.964344 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:47.966832 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.014027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.022066 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.025267 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.038810 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.047775 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.049386 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:48.132068 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:48.157828 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.158380 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.168155 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.171397 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.186374 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.187916 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.196179 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.196943 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.198493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.210041 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.210549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.216768 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.221352 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.221760 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.230134 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.230274 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:48.233891 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.242780 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:48.254307 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.256580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.259110 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:48.276432 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.276473 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:48.277675 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.283869 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.288422 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.304489 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.317520 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:48.373644 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.415194 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:01:48.420210 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:48.433171 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:48.445148 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.446676 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.448840 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.450268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.459852 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.463804 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.464152 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.473778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:48.484948 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.487349 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.493412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:48.504865 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.509538 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.522425 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.539927 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.541290 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.544557 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.549930 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.550901 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.551126 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.554376 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.565042 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.571345 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.571523 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.583082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:48.597147 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.615792 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:48.653182 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.695119 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.695397 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.703325 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.705307 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.718739 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.733433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:48.744848 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.803433 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.807137 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.809194 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.817077 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.820312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.820777 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.821986 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:48.826309 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:48.836082 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.855526 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.863976 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.878005 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:48.888947 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.908021 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:48.913318 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:48.939499 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:48.950837 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:48.954252 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:48.967078 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:48.968166 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:48.981739 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.000342 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.001269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.001292 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.011869 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.026276 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:49.055783 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:49.061190 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.061468 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.068943 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.069619 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.071782 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.081102 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.130226 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.133037 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.140697 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.145501 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.150483 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.178299 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:49.204624 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:49.209717 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.210791 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.218931 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.226658 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.234431 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.234747 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:49.294461 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.316727 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.317979 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.319309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.322605 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:49.330307 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.336412 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.336915 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.376163 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.389616 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.391173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.391924 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.396503 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.403578 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.417586 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.432394 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.437690 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.439158 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.442584 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.446085 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:49.453487 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:49.470438 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.490478 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.494313 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.502322 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.516075 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.516513 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:49.520781 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:49.521139 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:49.559188 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:49.576711 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.589974 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.595180 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.595312 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.599195 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.611178 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.611765 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.668696 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.670946 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.680420 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.682580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.689487 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.690784 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.731684 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.736706 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.736836 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.738395 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.741374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.743487 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:49.775097 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:49.790716 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.833391 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.840416 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.840641 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:49.844166 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.862221 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.866971 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:49.873596 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:49.906634 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.934782 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.936968 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:49.939893 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.941789 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.944437 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.945240 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:49.965989 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:49.973963 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:49.976199 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:49.987918 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:49.996526 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:49.999400 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.005424 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.011632 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.040012 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.043324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.044350 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.057456 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:50.096263 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:50.117375 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.117893 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.119362 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:50.130375 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:50.130558 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.132084 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.132617 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.181226 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.186235 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.187036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.192459 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.202577 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.215301 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.239365 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.242281 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.248474 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:50.250495 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:50.250576 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:50.251863 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.270349 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.282364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:50.295028 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:50.305747 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:50.306321 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:50.357939 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.375177 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.378032 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.390518 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.392060 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.392465 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.409009 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.412529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.421921 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.428771 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.431463 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.437690 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.438915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.446107 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.446480 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.450850 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.451742 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.490219 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.505246 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.509814 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.511757 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.527123 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.538761 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.539762 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:50.560735 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:50.566289 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:01:50.628345 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:50.634196 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.640802 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.643579 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.646482 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.651158 16338 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:50.651665 16338 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:50.652930 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:01:50.653059 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
I20240502 14:01:50.653818 16338 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:50.654047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:50.656327 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:50.656419 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:50.656458 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:50.656714 16338 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:50.682521 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.688616 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.688616 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.697538 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.702575 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.705955 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:50.720794 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.721860 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:50.738577 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.750144 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.750588 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.765218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.770725 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.784684 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:50.785071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:50.789359 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.799502 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:50.836123 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:50.861863 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.864521 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.866356 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.883208 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.894699 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:50.906455 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:50.916119 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.921833 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.925979 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.936436 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:50.941174 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.946501 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:50.947232 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:50.953660 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:01:50.958060 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:50.961822 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.970656 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:50.973539 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:50.985445 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.004732 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:51.031714 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:51.104902 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.115083 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.116503 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.123028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.123263 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.124887 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.152676 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.157488 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.159399 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:51.176132 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:01:51.178750 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:51.184901 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:51.199681 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:01:51.202811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:51.204241 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 14:01:51.208844 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:51.209573 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.212585 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:51.226389 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:51.235824 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.240358 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.254617 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.265192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:51.280061 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:51.281730 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.292493 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.297811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.298305 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.304643 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.309949 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:51.310139 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:01:51.324347 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.328150 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.332998 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.346148 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:51.369150 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:51.372277 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.372334 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.372977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:51.374229 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:51.448802 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.452117 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.452193 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:51.456346 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:51.459254 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.471275 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.496472 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.501498 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.516513 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.520875 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.531452 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.536798 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.537902 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.546447 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.550276 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.555194 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.561977 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:51.583514 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:51.590806 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.599043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:51.678709 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:01:51.689431 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.690340 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.692726 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.699630 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.703264 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.704408 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.704625 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.705760 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.712615 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.716143 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.718650 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.720505 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:51.751451 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:51.807297 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:51.864770 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.876343 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.901129 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:51.914873 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:51.922379 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.940726 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.943150 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.951218 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:51.959642 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.964920 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:51.967300 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.972479 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.974157 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:51.978690 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:51.979607 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:51.980577 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:51.980986 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:51.982888 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.023394 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.029495 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.031917 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.034193 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.041033 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.085114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.088387 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.097604 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.108027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.121472 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.122068 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.133833 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:52.140940 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:52.144644 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.151844 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.151902 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:52.205793 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.221246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.226032 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.228327 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.237915 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.242193 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.262085 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.264577 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.265415 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.270926 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.270951 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.280862 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.288522 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:52.290725 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:01:52.357203 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.359933 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.363530 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:52.376089 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.380038 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.380381 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.384538 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:52.416311 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:52.417061 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.417730 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:52.435312 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:52.436393 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.438102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.440497 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.448165 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:52.456444 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:52.460947 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:52.495518 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.505079 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.506706 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.511245 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.527158 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.530357 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.534065 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.535509 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.540428 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.551800 17852 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:52.551921 17852 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:52.552440 17852 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:52.553581 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:52.553720 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:01:52.554371 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:52.554472 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:52.554507 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:52.554648 17368 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:52.565532 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.585794 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.596473 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:52.637557 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.655215 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:52.661141 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.661773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.667052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.675289 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.697955 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.697937 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:52.723232 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:52.725564 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.728193 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.739724 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.748796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.760895 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.764726 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.786656 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.801733 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.801962 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.815101 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.830354 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.847692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:52.850030 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.851718 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:52.889519 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:52.890691 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.891533 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.894026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:52.904495 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.904799 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.906656 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.906674 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.907531 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:52.908134 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:52.914269 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:01:52.933487 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.938768 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:52.965850 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:52.992383 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:52.993680 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:52.995079 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.004308 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.006124 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:53.025641 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:53.028081 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:53.077970 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.084718 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.085007 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.085814 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.089231 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.093945 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.136770 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.137506 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.144711 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.147169 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.149366 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.156251 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.157030 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.225796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.230382 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.232249 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.234891 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.238467 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.259647 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:53.313499 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:53.314877 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.321175 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.321801 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:53.322738 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.330950 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.332751 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:53.340885 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.345376 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.354244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.361137 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:53.367293 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.368492 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.374112 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.380283 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.388530 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.391597 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:53.405301 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:01:53.448632 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:53.474535 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.477301 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.477504 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.480862 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.485659 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.501237 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:53.510726 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.512337 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:53.521425 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.534619 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.558912 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.561934 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.562937 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.577091 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.588145 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:53.595649 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:53.596536 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.601517 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.602807 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.622527 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.626837 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.627080 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.627646 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.644450 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.664268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.677070 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.677363 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.698247 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.702792 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.706559 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.713532 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.725701 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.753332 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.755024 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.756630 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:53.777848 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:53.811918 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:53.889873 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.902122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.904418 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.913519 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:53.920449 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.926846 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.932334 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.935822 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.939311 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.941251 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.949555 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:53.954717 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:53.973558 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:53.974766 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:53.975092 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:53.979962 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:53.995398 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:53.996344 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.002218 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.027040 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.033743 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.042250 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.049691 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.050200 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.064118 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.116773 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.118392 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.120820 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.125773 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:54.126950 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:54.148376 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.161370 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:54.163892 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:54.187885 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.191967 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.202250 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.217031 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.227875 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.232547 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.274451 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.280455 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.284580 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.289734 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.302814 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.311982 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.325546 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.337528 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.348450 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.351030 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:54.356624 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:54.366410 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.367883 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.438457 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.447722 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.450114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.455274 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.461422 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.462985 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.466367 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.474570 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.477651 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.480168 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.490962 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.504261 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.517782 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.534431 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.539400 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.540954 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.542733 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.545030 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:54.570258 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:54.593209 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:54.595990 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:54.598835 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:54.680776 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.698093 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.698745 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.699239 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.714658 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.720963 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.722898 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.724543 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.725569 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.725771 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.769142 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.772451 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.782907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.786836 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.797044 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.798282 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.805330 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.813547 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.815888 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.825592 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:54.843117 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:54.848852 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.850461 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.858714 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:54.865593 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:01:54.881462 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:54.951923 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:54.956849 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:54.958272 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:54.964825 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:54.976930 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:54.978555 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:54.994058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.001904 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.005472 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.011940 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.028585 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.031867 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.033159 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:55.053630 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.055269 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.061758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.062744 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.068744 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.075627 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.082216 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:55.096093 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.121801 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.175169 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.176476 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.181105 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.183547 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:55.186771 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.206624 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:55.215396 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:55.222039 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:55.240187 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.246294 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.246928 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.252548 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.252751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:55.262380 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.273499 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.288765 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.295341 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.298568 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.299185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.316749 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.317987 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.331319 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:55.368808 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:55.377485 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.378659 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.382381 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.383795 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.386646 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.408391 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.467263 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:55.478570 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.479385 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.490202 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.501245 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.506886 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:55.519080 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.519300 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.519546 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.532832 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:55.542160 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:01:55.542964 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.548418 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.554670 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.579535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.608393 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.610728 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.616487 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.629979 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.645326 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:55.647915 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:55.663054 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.678218 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.685428 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.686277 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.694669 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.707799 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:55.734463 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.739065 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.741354 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.745774 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:55.752578 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:55.756227 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.764638 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:55.767201 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.778203 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:55.792382 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.796527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:55.797319 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:01:55.798666 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:55.815568 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:55.853380 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.859624 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.862354 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.866370 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.868788 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.873019 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:55.878849 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:55.884853 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:55.885092 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:55.899402 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:55.921525 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:55.928601 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:55.933113 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.016942 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.018236 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.022435 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.025177 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:56.028788 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:56.029400 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:56.031883 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.037173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.038379 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.040510 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.042569 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.045414 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.067057 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.089740 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:56.111908 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.157960 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.165170 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.167829 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.182811 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.183602 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.187446 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.190075 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.192284 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.192477 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.195551 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:56.204555 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240502 14:01:56.206408 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240502 14:01:56.206912 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.212339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.212996 18962 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:01:56.213110 18962 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:01:56.213568 18962 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:01:56.214092 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:01:56.214221 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:01:56.227051 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:56.227176 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:56.227221 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:56.227346 18962 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:01:56.250452 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:56.303335 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:01:56.304210 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:56.322183 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.323019 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.326620 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.331341 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.334456 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:56.355031 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:56.366184 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.376374 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.382751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.391022 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.391927 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:56.394233 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:01:56.405342 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.416551 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.417925 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.425858 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.429934 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.465502 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.469522 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.486914 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.501713 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.520102 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.521986 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.527784 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.545279 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.546945 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.570602 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:56.576330 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:56.579741 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.590258 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.590346 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.599282 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.608688 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:56.630074 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.662273 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:56.677428 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.681643 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.698217 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.701735 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.703511 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.704519 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.710932 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.729249 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.746469 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.749626 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:56.760998 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:56.777945 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:01:56.790997 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.792996 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:56.803797 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:56.813017 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.814383 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.821565 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.824738 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:56.834511 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.865948 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.866888 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:56.876706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:56.884151 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:56.886447 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:56.905376 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:56.917348 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.016609 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.026362 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.026793 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.031221 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.039036 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.039340 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.055480 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.058404 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.065672 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.066653 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.067994 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.085477 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.085530 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:57.107372 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:57.108741 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.125192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.128067 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.128423 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.129266 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.151824 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.180223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.184022 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.190004 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.190877 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.193101 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.209203 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.233848 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:57.258589 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:57.263178 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.266695 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.286999 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.292436 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.311269 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.324930 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.348882 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.349948 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.352540 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.363370 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.364487 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.381269 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:57.409602 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.415688 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.425495 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.425527 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.425793 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.426203 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.430639 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:57.450188 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:01:57.462790 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:57.483132 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:57.504400 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.506770 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:57.522699 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:57.525594 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.532130 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.534902 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.550271 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.553507 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.553826 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.558830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.578791 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.584656 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.604487 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.616443 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:57.622510 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:01:57.654075 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.665864 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.681301 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.681493 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.694404 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.701666 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.705308 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:57.712116 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:57.715574 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.718673 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.721691 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.728669 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.730950 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.752691 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.795764 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.800925 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.812531 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.835526 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.839179 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.844897 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.845669 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.847465 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.855020 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.856552 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.856774 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:57.877305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:57.883791 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.896607 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:57.919335 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:01:57.946326 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:57.959056 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:57.959594 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:57.960100 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:57.961216 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:57.964524 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:57.964881 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:57.982705 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.064891 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.071442 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.073817 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.075268 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:58.083315 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.112898 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.113536 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.116194 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:58.123641 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.127519 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.131139 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.134673 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.137665 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.138321 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:58.143944 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:01:58.144748 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.157339 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.159327 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.160349 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.163515 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:58.172473 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.211382 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:58.213907 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:58.255178 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.261943 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.267275 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.282968 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.283135 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:58.291541 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.302511 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.355947 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.357463 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.369141 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.385385 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.388056 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.388803 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.400277 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:58.400617 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.410208 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.419505 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.427901 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.430634 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.432807 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.450197 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.454779 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.461912 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:58.463212 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.466454 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.469029 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:58.478305 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.488380 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.533435 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.539680 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.554409 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:58.559087 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.563877 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:58.570690 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:58.572461 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.580905 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:58.604015 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.634539 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.637249 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.637678 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.641263 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.645057 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:58.662168 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.666811 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.704001 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.714378 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.725678 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.730204 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:58.732687 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:58.755875 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.758555 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:01:58.767331 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:58.790431 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.801975 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.804093 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.808554 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.819185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.836869 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.897639 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.912123 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.913532 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:58.915874 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.917133 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.920403 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.927423 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:58.930254 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.935932 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.936421 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.941131 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:58.949945 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.960887 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:58.976845 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:58.995353 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:58.997354 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.001233 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.013521 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:59.040566 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:01:59.047183 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:59.093515 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.097142 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.101092 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.111196 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.112035 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.138284 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.184491 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.192153 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.195906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.202652 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:59.204214 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:59.205192 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.205283 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.216742 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.228457 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:59.230232 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:59.231488 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.240491 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.254446 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:59.264725 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:59.266995 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.299690 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.304198 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.319913 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.322535 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.336681 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.341270 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.345985 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:59.350986 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:59.359448 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.366472 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.366864 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.371063 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.376685 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.444770 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.451015 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.455071 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.466588 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.477561 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.486523 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.533931 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.537202 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.542084 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.550276 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.555465 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.563724 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.578713 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.582757 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.584228 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.593469 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:01:59.604661 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:01:59.608507 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.627995 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:01:59.703541 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:59.709156 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.709390 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:01:59.714205 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:01:59.729389 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.729530 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.741541 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.780690 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.789197 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.798509 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.798921 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.799031 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.807822 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.809415 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:01:59.809564 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:01:59.821686 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:59.824368 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:01:59.829291 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.834733 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:01:59.842037 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.850931 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.873452 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:01:59.873930 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:01:59.877455 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.880524 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:01:59.885411 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:01:59.892788 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:01:59.912506 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:01:59.985229 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:00.022596 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.025796 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:00.032589 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.039849 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.040253 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.041241 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.044626 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.050242 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.055670 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.060421 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.063262 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.072388 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.076647 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.094565 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.100045 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:00.101070 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.111138 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.112463 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.114341 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.129496 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:00.138756 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:02:00.183089 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:02:00.227797 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:00.230006 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.235795 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.269551 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.277429 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.282354 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.302130 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.307782 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.308310 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.312413 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.316746 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.328539 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.340412 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.347201 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.350823 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.360136 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:00.363469 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.367321 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.369354 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.375536 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.388571 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.392072 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.401185 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.402062 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.403791 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.414536 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.415983 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.420490 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:00.495891 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:00.524314 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.529803 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.531800 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.544495 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:00.545099 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:00.554714 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.555227 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.567521 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:00.568179 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.577527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.583602 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.588701 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.607764 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:00.608439 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.614643 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.641343 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.642464 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.646026 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.655297 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.655922 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.675936 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:00.712937 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:00.723552 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.723670 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.727420 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.732918 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.756861 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.761301 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.806123 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:00.819664 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:00.825402 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:00.833047 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:02:00.846109 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.862819 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.865682 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.880589 21564 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:00.880694 21564 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:00.881140 21564 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:00.882040 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:00.882160 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
I20240502 14:02:00.882750 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:00.883824 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:00.883944 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:00.883986 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:00.884343 21564 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:00.885274 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.890525 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.895675 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:00.897485 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.899227 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.903151 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.905633 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:00.905966 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:00.912359 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:00.931588 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.939492 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:00.942234 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:00.950865 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:00.967187 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.025918 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.030606 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.035326 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:01.038337 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.053911 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.063831 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.079368 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.082859 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.091236 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:01.091809 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.097110 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.098594 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.131611 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.139940 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.148635 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.158891 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.163049 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.180963 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.182413 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:01.191879 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.193689 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.193768 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.194799 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.196380 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.199934 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:01.209976 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 95 similar messages]
I20240502 14:02:01.211671 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.213501 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:01.258558 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:02:01.259212 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.280769 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:01.285558 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:01.312572 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.317660 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.323025 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.328420 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.331727 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.333195 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.342998 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.351011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.352411 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.363031 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.363435 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.366561 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.389173 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:01.412196 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 14:02:01.456211 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.466634 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.469476 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.471773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.499564 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.500674 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.505982 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.512748 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:01.515580 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.529597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.541590 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.541730 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.552006 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:01.565421 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:02:01.573364 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:01.589613 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.656258 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.657989 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.662454 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:01.671743 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:01.673117 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:01.674819 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:01.675453 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.675822 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.687947 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:01.698777 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:01.702616 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.715243 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.735055 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.740564 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.759199 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:01.784178 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:02:01.790557 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.803948 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.804057 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:01.805693 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:01.810775 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.823312 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.827369 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.904014 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.910533 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.915278 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.928361 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:01.940250 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:01.944615 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.953085 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:01.956859 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.959360 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:01.965237 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:01.980607 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:01.999787 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.000751 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.009780 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.013500 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.022183 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.026247 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:02.033190 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.056138 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.075011 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.075029 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.083532 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.106029 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.106671 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.115087 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.200296 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.203052 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.205794 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.206271 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.209254 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.210582 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.212006 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:02.232211 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:02.239060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.241091 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.264719 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.266964 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.271960 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:02.297353 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:02.307962 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.325178 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.330981 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.339092 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.340791 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.341233 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.346340 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:02.362496 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.367381 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.382668 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.382680 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.389820 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:02.409076 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:02.409188 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:02.447149 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.455482 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.460906 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.461180 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.478097 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.480160 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.480787 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:02.513916 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.530543 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.531086 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.536216 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.542030 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.550197 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.556919 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:02.560376 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:02.608649 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.620545 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.626129 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.629918 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.630023 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:02.636564 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20240502 14:02:02.636710 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:02.669672 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:02:02.696946 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.704897 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.710100 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.737025 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.750411 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.759924 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.764403 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.766247 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.766691 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.775403 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.777431 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:02.804716 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:02.808598 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.813704 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.845007 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.852178 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:02.857473 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.870283 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:02.870956 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:02.877501 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.894388 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:02.898013 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:02.902735 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.903594 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:02.904428 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:02.907706 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:02.942768 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:02.980587 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.002724 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:03.011852 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.015501 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.016914 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.043170 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.051645 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.058543 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.061057 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.061969 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:03.064442 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:03.070487 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.070808 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.079397 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.095252 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.105878 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:03.151952 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.159941 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:03.160953 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.163023 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.166668 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.170750 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.181450 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.242687 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.247689 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.248639 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.250312 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.265480 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.265496 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.289765 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.310796 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.324661 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.330363 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.331476 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.339715 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.341884 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.360332 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.361653 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.362228 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.371050 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.374714 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.386314 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.405442 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:03.410115 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:03.418342 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.476068 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.502707 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.515355 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.521198 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:03.522548 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:02:03.527137 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.529150 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.539737 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.554435 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.555516 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.555938 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.560985 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.561244 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.570884 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.583086 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.584306 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.585189 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.594110 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.597175 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.611550 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.611927 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:03.645306 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:03.646669 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.652951 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.670884 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.672803 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.679589 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.680410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.680717 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:03.704520 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.735679 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:03.746400 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.749256 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.751063 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.751894 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.752264 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.752450 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.756759 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:03.786657 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:03.837121 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.842566 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.849578 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.850832 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:03.861640 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:03.865065 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.865069 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:03.914990 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:03.922106 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.923803 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.924258 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.938491 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:03.946799 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:03.949015 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:03.977820 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:03.978753 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:03.983592 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:03.994272 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:03.997954 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:04.012777 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.026253 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.032189 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.032425 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.036249 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.036330 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:04.040802 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:04.043184 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.054152 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.136835 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.136934 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.138002 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.144927 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.145334 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.154474 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.182066 23733 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:04.182231 23733 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:04.182772 23733 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:04.183369 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:04.183521 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:02:04.186905 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:04.187043 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:04.187114 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:04.187649 23733 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:04.211107 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.212697 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.220374 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.222833 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:04.234365 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:04.236208 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:04.239603 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:04.264559 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.321545 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.327383 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.337491 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.338879 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.339877 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.341154 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.356442 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.360898 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.365881 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.374078 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.374979 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.381947 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.476658 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.477664 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:04.482689 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:04.483328 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.495764 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.517167 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.528463 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.530519 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.539052 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.543361 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.561347 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.566227 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.570080 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.571416 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.599376 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.602373 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.616441 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.616492 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.620887 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:04.623833 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:04.624228 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.627944 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.629973 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.642391 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.654650 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.662799 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:04.735939 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:04.764843 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:04.811857 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.816602 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:04.817281 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:04.819011 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.841861 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:04.844995 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:04.853188 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.856480 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:04.859221 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.867193 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.872514 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:04.876354 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.895462 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:04.926234 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.951862 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:04.963626 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:04.967068 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:04.980024 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:04.990451 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:05.001322 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:05.004101 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:05.012552 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.017547 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:05.054471 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:05.064942 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.088575 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.092787 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.097036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.098744 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.107388 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.127792 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.148506 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.162933 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.165230 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.166682 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.168989 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.191359 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:05.203984 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:02:05.210485 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.229221 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.230283 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.234853 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.235729 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.249229 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.250368 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:05.257268 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:05.300382 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.301035 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.301476 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:05.312373 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.315204 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.315296 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.332495 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.403970 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.406857 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.413735 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.415887 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.421394 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:05.439956 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:05.441906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.442226 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.442994 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.451388 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.451704 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.456238 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.471863 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.486687 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.491144 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:05.492770 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:05.493752 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.496439 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.506289 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.511164 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.559598 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.569716 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.574049 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.574112 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.591938 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.604986 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:05.610972 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:05.618772 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:05.662932 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.670878 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.671519 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.675015 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.677250 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.683665 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.711861 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.721581 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.725988 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.726965 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:05.730538 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:02:05.742439 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.744057 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.762977 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.794134 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.807329 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.809453 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:05.810693 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.811955 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.828812 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.890532 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.906877 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:05.907061 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:05.907959 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:05.910611 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:05.930994 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:05.954941 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:05.988442 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:05.988592 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:05.999152 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.004191 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.006932 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.010331 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.012164 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.013515 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.016662 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:06.029690 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:06.030488 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.051636 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.053262 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:06.053598 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.062157 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.107105 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.114327 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.115749 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.124889 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.126258 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.126775 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:06.144979 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:06.159410 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.204491 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.205448 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.207641 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.218981 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.221916 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.227104 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:06.236948 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.240798 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:06.243492 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.246693 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:06.248992 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:02:06.250134 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.252943 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.264348 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.272667 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.290127 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:06.311437 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:06.320484 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:02:06.321431 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.342272 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.342319 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.343724 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.344838 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.362792 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.364665 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.365443 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.380370 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.393947 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.396832 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.412364 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:06.424465 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.477202 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.479044 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.483352 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.500615 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.502851 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.512346 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.512802 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.513623 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.524240 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:06.524904 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:02:06.528908 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.538079 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.540053 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:06.556900 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:06.583065 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:02:06.583679 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:06.605567 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:06.626083 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.629349 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.635376 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.638648 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.664981 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:06.667448 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.681807 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.684479 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.685134 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.691177 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.691592 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.697252 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.702816 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.764711 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.765635 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.771490 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.781458 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.785439 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:06.802526 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:06.806192 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.839424 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:06.840624 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:02:06.850644 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:06.862939 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.866047 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.878625 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.887478 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.920719 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:06.926240 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:06.932104 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:06.948267 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:06.948418 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:06.951140 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.008606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.013610 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.019862 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.023398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.037106 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.042335 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.049381 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.060029 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.077755 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.090637 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.091919 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:07.093127 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:07.094278 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.127517 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.130692 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.133445 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.144418 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.149313 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.156762 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:07.158051 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:07.213541 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.219187 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.220997 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.233583 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:07.242763 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.244138 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:07.251875 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:07.255909 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:07.261768 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.302757 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.308216 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.311161 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.321259 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.340669 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.345484 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.375799 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:07.389955 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.394181 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.402951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.403290 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.406948 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.434698 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.474305 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.477504 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.477670 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.488092 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.489532 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.521869 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:07.524910 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:07.526154 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.527657 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:07.533432 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:07.541290 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:07.546661 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.566660 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.568506 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.581197 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:07.597136 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:07.604934 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:07.608448 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.612926 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.625488 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.626840 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.628468 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.639137 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:07.654551 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.669716 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.670056 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.674069 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.687420 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.687690 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.697947 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:07.705439 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:07.762023 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.788933 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.794543 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.797284 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.819989 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.820266 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.828492 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.867070 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.867861 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.867877 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.876515 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.876614 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:07.879438 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.902909 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.907145 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.907738 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:07.932066 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.932456 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:07.932677 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:02:07.961339 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:07.970811 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:07.984239 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:07.988639 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:07.988909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:07.994382 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:07.995450 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.014418 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:08.033793 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:08.049440 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:02:08.052130 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 14:02:08.067160 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.071328 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.073410 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.084376 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.084992 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:08.096345 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.114023 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:08.114931 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:08.122809 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:08.125974 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:08.143774 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.143819 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.147883 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.157187 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.178216 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.196106 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.200680 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.202673 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.209034 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.222453 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.224396 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.301229 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.311717 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.320076 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.323336 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.324586 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.346715 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.390986 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.401693 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:08.402076 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.407008 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.408991 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.409451 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.411091 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.423696 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.431229 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.453388 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.455273 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.457075 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.461004 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:08.496845 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.497157 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.501917 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.504910 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.513372 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.513506 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.513907 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:08.535930 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:08.555689 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:08.557945 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:08.583258 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:08.591645 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:08.595863 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:08.597791 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.599632 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.600318 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:08.623503 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.624436 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.629168 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.657223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.660529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.663389 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.664744 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:08.665436 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.673846 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.676868 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:08.720518 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:08.731050 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:08.754776 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.764402 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.768208 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.768944 26122 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:08.769060 26122 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:08.769693 26122 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:08.770109 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:02:08.770239 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:02:08.772089 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:08.777298 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:08.777397 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:08.777431 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:08.777750 26122 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:08.792673 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.811096 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.854409 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.870673 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.875854 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.875926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.877653 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.879274 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.901783 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.903858 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.905485 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:08.930094 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:08.942462 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.954708 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:08.972998 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:08.985128 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:08.994364 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:08.999642 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.000934 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.001194 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:09.011169 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:09.045823 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.052363 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.057219 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.057781 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.062906 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.071229 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.182821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.183784 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:09.196635 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:09.198691 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.201128 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.225458 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.226747 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.228415 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.230760 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.231412 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.238593 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:09.254864 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:09.255803 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.256780 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:09.264879 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:09.265511 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.275506 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.280529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.286135 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.293938 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.302780 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.357161 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.373075 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.373345 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.374225 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.378705 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.403745 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:09.441743 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:09.474215 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:09.482307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.487315 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.488150 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.489929 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.513484 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.518846 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.520684 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.532460 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.537475 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.543746 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.552839 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.559938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.560478 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:09.560807 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:09.561674 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.571624 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.580633 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.616892 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.620074 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:09.623667 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:09.642799 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.645565 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.649147 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.651796 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.656116 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.668411 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.745527 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.750267 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.755101 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.760429 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.768451 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.770681 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:09.807472 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:02:09.827306 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:09.830811 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:09.830902 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:09.831765 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.850646 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.857776 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:09.858029 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.859967 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:09.868321 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:09.868980 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.868985 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.879635 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:09.883103 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:09.886529 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:09.888425 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:09.943018 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:09.970033 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:09.971771 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:09.982939 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.002171 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.003556 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.007145 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.020257 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.031487 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.041711 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.046833 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.047310 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.057145 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.100361 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.104373 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.106011 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.109364 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.110275 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.122627 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:10.173286 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:10.187901 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:02:10.203354 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:10.211562 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:10.229342 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:10.238951 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.245707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:10.257949 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:10.263039 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:10.277478 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:10.279752 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.285702 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.289467 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.289932 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.290215 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.290304 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.300451 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:10.302294 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:02:10.313050 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.317267 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.329470 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.333959 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.351158 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.364115 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.364362 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.365869 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.370496 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.372788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.381218 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.381574 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.389905 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.398365 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.420104 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:10.434002 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:02:10.492146 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:10.502358 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.503612 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.515277 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.527058 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.531109 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.550423 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.556428 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.567287 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.567705 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.576525 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.579800 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.608412 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.610177 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.617230 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.617812 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.618026 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:10.625427 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:10.629817 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.654928 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:10.683730 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:10.708137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.709210 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.715763 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.718942 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.722467 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.741251 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:10.795872 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:10.802357 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.805605 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.812523 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.823309 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.825004 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.838551 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:10.860196 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:10.908828 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.917685 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:10.922943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:10.937281 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.944445 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:10.949443 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:10.949745 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:10.957653 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:10.966392 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:10.969419 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:10.992715 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:10.997256 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:10.998241 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.014246 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:11.032629 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.055603 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.065518 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:11.077636 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:11.083042 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:11.085094 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.091704 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.104804 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.114189 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.120433 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.127867 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.130223 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.149995 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.180397 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.187424 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.187695 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:11.190055 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.195216 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:11.199515 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.199828 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.201625 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.202044 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.225050 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:11.276422 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.282007 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.292882 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.297530 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.298362 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.301960 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.304900 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:11.318398 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:02:11.332517 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:11.346148 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.347326 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.352859 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.357404 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.359830 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.373944 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:11.385399 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:02:11.416954 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.424263 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.429657 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.431787 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.439777 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.446696 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.449872 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.471486 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.477303 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.478091 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.478510 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.479235 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:11.509225 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.577955 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.583302 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.586582 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.620777 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.624492 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:11.630767 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:02:11.641680 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:11.651684 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:02:11.652333 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:11.653559 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.665356 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.678038 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.697355 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.698973 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.704443 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.757040 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.763144 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.772886 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.773316 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.789266 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.800329 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:11.815922 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:02:11.856869 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:11.857821 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:11.863445 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.866400 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.869822 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:11.882009 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:11.886087 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:11.886499 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.888023 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.895840 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:11.900069 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:11.901875 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:11.907640 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:11.913473 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:11.915891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.025789 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.029709 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.035604 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.042671 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:12.059101 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:12.061364 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.065122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.069784 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.078505 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.078696 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.082777 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.087278 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.108826 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.125067 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:02:12.127303 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:02:12.142596 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.146806 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.159821 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.161561 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.169873 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.182382 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:12.205940 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.212759 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.215787 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.216418 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.217043 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.222497 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.231045 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.308373 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.323506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.324290 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.326862 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:12.333457 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:12.340237 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:12.347772 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.348964 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.359676 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.396198 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.401526 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.401998 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.404615 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.413591 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.417174 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:12.461738 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:12.474478 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.477376 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.487838 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.495517 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.505230 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.506351 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:12.543116 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.543845 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:12.546849 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.562992 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.575773 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.576582 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.587960 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:12.603394 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.603744 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.609897 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.610430 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.611410 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.612499 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.628202 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.651666 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.661465 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.666983 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.667165 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.672027 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.689523 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:12.692147 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.702972 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.724442 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:12.726424 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.757841 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:12.774492 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.782285 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.793797 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.794418 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.800292 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.811923 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:12.817255 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:12.853696 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.887801 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.897758 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:12.897840 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.900274 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.908886 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.922626 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:12.933466 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:02:12.934885 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:12.938702 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:12.940030 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:12.953369 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:12.954098 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:12.959292 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:12.976989 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:13.014983 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:13.073720 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.082078 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.084555 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:13.092429 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.094866 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.098609 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.114239 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.123481 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.124085 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:13.135789 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.148463 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.154469 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.161186 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.162278 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.175941 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.186408 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.190584 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.191780 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.201210 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.211987 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.214311 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.217669 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.220779 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.222502 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.247001 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.252493 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.332583 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.334623 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.341820 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.342113 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.344942 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:13.355906 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.360917 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:13.399997 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.404673 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.406033 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.407447 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:13.408438 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:13.411933 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:13.429278 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:13.429626 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.430430 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:13.438771 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.469179 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.469322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.471824 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.479431 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.479645 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.508360 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:13.560804 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:02:13.564455 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.576380 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.581413 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.585672 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.590595 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.612915 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.632180 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.638614 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.641438 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.641943 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.641953 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.650780 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.654735 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:13.706336 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.709664 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.724938 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.733247 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.741029 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.746476 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.754714 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.756883 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.757216 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.759465 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.771045 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.773303 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:13.779872 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.785331 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:13.896266 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:13.898072 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.900453 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.915490 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.923002 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:13.931557 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:13.939283 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:13.947916 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.948259 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.954615 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:13.955941 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:13.958192 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:13.960224 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:13.960422 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:13.975203 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:13.979373 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:13.992056 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.011757 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.016077 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.016393 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.041422 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.064267 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.067822 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:14.076841 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:14.077594 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.088964 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.096300 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.096417 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:14.099769 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.173050 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.176934 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.177371 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.184072 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.187709 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.211925 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.240247 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.242043 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.247709 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:14.257467 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.269474 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.277746 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.278420 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.301409 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.311429 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.311676 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.322620 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.323679 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.350425 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:14.356191 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:14.401069 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.425540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.427003 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.427259 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.428311 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:14.456499 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.467695 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.471079 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.482856 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.483255 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.494931 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.495807 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.498150 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.511080 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:14.512645 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:14.521579 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.538724 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.547008 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.550220 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.556229 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.562578 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.569937 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.570645 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.600164 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.606609 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.610203 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.619838 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.715286 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.718070 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.719328 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.735954 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.738910 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.740494 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:14.776343 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:14.782558 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.810496 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.813221 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.829829 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.838343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:14.840348 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.840721 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:14.842628 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:14.842708 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.849671 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.864118 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.869974 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:14.871224 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:14.873142 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.873682 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:14.875429 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:14.901556 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:14.904156 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:14.915549 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.932580 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:14.934136 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:15.020260 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.044076 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:15.049528 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:15.059724 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.070340 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.070339 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.078603 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.081303 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.097698 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.099959 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.104127 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.109616 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.110229 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.110236 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:15.121688 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:15.130885 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.151057 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:15.151115 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.152119 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.155262 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.155876 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.158217 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.168980 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:15.183909 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:15.211577 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:02:15.247205 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.253926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.271167 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.274317 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.276418 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.289000 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:15.316998 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.326040 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.331960 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.337561 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.348485 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.356981 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:15.366664 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:15.372118 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.373405 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.386905 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.386926 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.388760 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.393975 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.408610 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.424938 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.473030 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.475494 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.482813 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.483148 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:15.485885 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:15.499409 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.503147 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:15.522008 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:02:15.585292 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:15.589398 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:15.594089 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:15.603986 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.619112 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.626863 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:15.629402 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:15.637780 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.644194 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.668013 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.677388 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.679828 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.681277 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.684576 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.694782 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.711254 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.712594 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.716130 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.736471 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.744830 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.762143 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:15.792323 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:15.818627 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:15.825055 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:15.842165 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.848845 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.850205 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.861414 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.866442 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.874732 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.931490 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.936883 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:15.936916 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.941350 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.948870 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:15.953747 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:15.956912 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:15.962157 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:15.986650 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:15.990777 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.011989 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.015511 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:16.057524 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:16.094441 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.094614 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.110761 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.129559 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:16.132102 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.141353 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.145844 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.147439 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:16.153538 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:16.162432 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.162716 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.165364 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.174675 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:16.190035 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.192973 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.215792 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:16.217516 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.218595 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.232676 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.240885 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.252640 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.254004 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.260424 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.260586 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.265862 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.275823 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.276156 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.291138 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.388618 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.397298 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.400032 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.401543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.408763 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.428743 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.451191 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.453711 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.454193 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.456449 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.456699 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.467074 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:16.473479 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.478574 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:16.487536 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:02:16.489039 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.494125 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.495930 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.496621 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.501410 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.503548 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:16.526470 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:02:16.541529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.542452 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.545912 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:16.558526 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:16.572782 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.584252 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.585598 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:16.621181 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:16.663272 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:16.682860 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:02:16.706151 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.709018 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.714835 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.715390 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.728808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.730453 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.734338 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.735023 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.744676 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.763732 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.764412 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.770192 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:16.771310 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:02:16.784632 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.793012 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.811813 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.812073 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:16.813668 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.819975 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.845149 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.857410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.860060 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.868045 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.868443 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:16.870237 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.872596 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:16.894825 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:16.943430 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:16.954373 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:16.954939 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:16.956346 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:16.957540 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:16.960569 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:16.979970 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.016336 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.017791 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.022588 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.022662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.030089 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:17.037545 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.048426 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:17.060621 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:17.095824 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.099717 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.103312 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.106880 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.108489 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:17.119776 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:17.126727 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:17.171480 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:02:17.183504 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:02:17.198163 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.203661 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:17.216434 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:17.229182 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.232461 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.238245 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.248488 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.262552 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.263268 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.265748 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.268819 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.282316 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.290223 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.292171 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.294399 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.298595 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.314538 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.329650 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.331800 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:17.422308 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:17.457561 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.482092 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.482793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.495386 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.504398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:17.506072 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.507122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.511225 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.548988 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.554654 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.554908 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.556186 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.565470 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.579921 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.584554 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.589671 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:17.592749 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.597199 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.599633 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.610126 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.613394 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.618891 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.619441 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.626941 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:17.628834 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.634734 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:17.650398 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:17.652197 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:17.663673 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:17.676190 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:17.692487 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:17.701215 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:17.715857 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.716058 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.720111 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.723105 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.723642 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.726280 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:17.763713 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.823191 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.825670 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.827414 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:17.829236 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.845970 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.848352 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.861613 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:17.870234 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:02:17.893594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.893846 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.912056 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.912941 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.915899 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:17.917078 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:17.921856 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:17.953356 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.958139 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:17.973696 32423 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:17.973811 32423 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:17.974228 32423 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:17.975543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:17.975759 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:02:17.975916 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:02:17.984092 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:17.990511 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:17.995832 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:17.997332 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.000970 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.010437 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.014654 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.024735 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:18.025107 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:18.025235 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:18.025291 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:18.025507 32423 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:18.030218 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.103328 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.110814 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.115886 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.122773 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.124033 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.129135 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:18.159653 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:18.191543 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.200335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.209861 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.238205 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:18.242018 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:18.247515 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.251638 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.253687 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.256807 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.266947 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.282147 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.286062 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.287345 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.293789 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.295182 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.300309 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.309266 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.316401 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:18.326243 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.326771 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.330729 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.421677 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.423914 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.427094 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.432077 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.440326 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:18.461259 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.466956 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.481953 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:18.489996 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.505306 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.508716 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.511907 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.525666 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.528452 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.530262 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.538012 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.542047 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.543551 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.559700 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.559970 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.563098 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.570909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.578539 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.580430 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.585848 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.603991 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:18.619951 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:18.639971 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:02:18.643604 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:18.695236 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:18.706252 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.712636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.715773 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.723472 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.724949 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.735913 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.742065 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.748507 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.767830 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.776767 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:18.777281 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:18.778380 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.788947 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:18.809340 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.879406 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.880518 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.882613 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.886572 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:18.900758 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:18.907392 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.920010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:18.944420 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:18.944991 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:18.946322 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:18.948199 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:18.953059 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:18.978056 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.038899 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:19.039769 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.044019 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.049942 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.060768 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.064841 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.081888 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.094527 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.094535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.108942 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.118189 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.118332 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.140780 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.167434 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.168291 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:19.180680 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:19.181500 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.182461 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.194283 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.194542 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:19.195278 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:19.211421 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:19.242017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.256819 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.259795 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.265110 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.274292 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.301426 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.323467 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.336798 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:19.337965 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.349776 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.354789 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.355013 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.372431 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.379781 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.385985 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.386974 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.393368 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.406476 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.415074 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.454280 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.457406 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.466017 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.468386 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.477800 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.485592 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:19.523797 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.545845 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.552503 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.555478 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.570737 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.571708 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.576409 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.586199 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:19.595582 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:19.609367 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.611527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.612008 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.621526 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.653911 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:19.694331 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:19.705302 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.715903 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.726768 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.731197 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:19.753103 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.763098 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.770311 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.771673 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:19.780876 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:19.785034 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.791088 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.791937 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:19.797951 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:19.798048 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:19.817329 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.827576 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:19.832099 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:19.834888 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:19.837338 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:19.853509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:19.853509 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:19.853610 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.857371 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:19.883189 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:19.888247 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:19.971462 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:02:19.998428 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.001560 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.008610 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.010921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.026422 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.047331 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.049408 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.066220 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.066504 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.069242 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.079279 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.089533 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.092523 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.092736 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.101717 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.105834 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.110435 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:20.111622 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:20.113852 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:20.118991 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:20.134912 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.136168 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.157119 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.157510 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.161546 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.166528 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.169553 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.175616 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.177014 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.178615 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.183003 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:20.189378 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:20.192004 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.193017 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:20.194075 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:20.249619 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:20.253548 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:20.269977 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:02:20.313057 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:20.315363 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.318710 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.325664 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.341457 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.349920 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.352890 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.358150 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.373170 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.376508 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.377409 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.383780 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.388248 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.388525 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.396121 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.412499 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.413583 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.414106 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.425370 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:20.437662 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:20.549357 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:20.550096 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.551432 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.554730 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.569840 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.572485 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.577704 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.583842 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.589401 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.592324 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.598134 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.598768 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.628696 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.634356 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:20.648866 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:20.651098 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.653368 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.669371 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.671996 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.679267 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.780376 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.795675 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.803098 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.805889 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.807034 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:20.814189 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.818493 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.826712 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.828001 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:20.836735 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:20.837222 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.840229 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.845800 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:20.879763 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:20.929220 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:20.932792 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:20.937865 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:20.941288 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:20.942008 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:20.958849 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:20.961050 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:20.973973 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:20.999047 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.002192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.005203 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.007812 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.022655 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:21.034958 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:21.038691 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.042258 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.044644 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.057423 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.060395 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.066385 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:21.076524 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:21.081393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.128396 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.132601 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.132625 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.133392 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.135818 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.140727 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.190665 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.192673 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.195477 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.199019 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.204684 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.219386 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:21.291316 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:21.306403 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.310643 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.319353 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.319388 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.321128 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.322197 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.352770 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.376475 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.379117 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.384431 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.384974 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.395157 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:21.437018 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:21.457571 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:21.477288 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.483609 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.487811 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.501272 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:21.511379 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:21.513214 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.519635 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:21.546983 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:02:21.547432 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:21.548303 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:21.555384 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.559091 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.566922 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.577678 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.577816 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:21.585637 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:02:21.620386 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.624223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.624537 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.624951 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.626219 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.633090 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:21.636793 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:21.673866 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.674481 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.674599 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.675540 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.695080 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.697881 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.710628 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.712520 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.719512 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.725982 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:21.741297 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:02:21.745754 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.748395 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:21.823350 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:21.852545 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:21.862180 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.863787 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.871690 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.872462 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.888661 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:21.890864 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240502 14:02:21.897143 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:21.901157 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.913681 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.918996 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.921706 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:21.933647 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:21.944988 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.970084 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:21.973093 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:21.977063 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:21.989835 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:21.993956 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.004179 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.009356 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.029461 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.040537 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.042444 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.056227 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.056890 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:22.065181 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.071508 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.106247 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.109113 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.110733 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.115406 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.119390 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.129674 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:22.150846 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:22.155700 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:22.165910 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.204149 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.205942 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.207392 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.209323 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.215162 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.215200 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.216361 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.219451 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.222385 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.232641 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:02:22.233827 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.256162 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.256631 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:22.316673 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:22.349339 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:22.362007 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.363608 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.371963 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:22.373324 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.373705 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.374089 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.379794 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.383383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.383975 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.402673 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.402957 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:22.405754 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.410825 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.417661 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.425748 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.435801 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.472098 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:22.502522 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.502763 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:22.503502 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.504062 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.519248 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.526019 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.528820 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.538000 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:22.588572 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.603199 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.604200 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.609346 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.613250 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:22.625357 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:22.629432 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.641765 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.643898 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.645999 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.650302 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.653568 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.667692 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:22.693158 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:22.697081 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.723407 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.734586 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.734781 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.740597 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.749518 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:22.750808 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.757937 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:22.850095 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:02:22.855001 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:22.870189 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.874159 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.874411 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.880476 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.888471 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.896893 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.899061 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:22.910250 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:22.912963 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.914124 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:22.917356 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.918004 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.929257 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:22.936161 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.936138 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:22.945637 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:22.968976 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:22.972399 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:22.993105 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.012646 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.013463 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.020882 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.021260 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.022876 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.033888 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:23.056231 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:02:23.059423 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.065721 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.072907 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.076107 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.080578 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.112520 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.170465 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.172407 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.176194 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.176753 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.177784 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.180965 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.236564 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:23.237905 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:23.241230 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.244607 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.245949 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.254837 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:23.268067 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:23.275265 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:23.276875 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:23.289131 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:23.330070 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:23.330808 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:23.332464 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.338035 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.345885 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.359166 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.376706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.415367 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.418740 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.420681 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.422910 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.424038 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.431878 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:23.435369 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:23.492331 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.495239 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.501940 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.502717 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.519220 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.521061 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.533922 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.537353 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.554008 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.578336 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.579504 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.580395 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.581060 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.582266 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.582490 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.603597 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.606572 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.627722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:23.643214 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:02:23.643939 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:23.714701 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.721297 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.721880 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.727097 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.730314 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.736478 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:23.778088 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:23.784822 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:23.793092 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:23.794029 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.810354 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.816699 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.822119 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.833623 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:23.855701 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:23.867050 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.872659 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.883229 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.892661 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.897202 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.903684 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.907765 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.908574 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.909739 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.936261 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:23.944205 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.944705 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:23.946326 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.964715 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:23.966102 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:23.971396 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:23.984244 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:23.985054 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:24.018571 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:24.019086 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.043007 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.050729 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.052171 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.064023 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.068972 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.082949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.134693 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.137204 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.139500 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.142485 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.153729 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.166913 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.205214 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.208367 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.217129 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.217692 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:24.226212 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:24.230894 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.249961 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:24.258765 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:24.270552 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:24.278241 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.310420 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.313927 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.315241 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.333045 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.347819 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.363811 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.404480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.415964 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.432497 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.442633 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.443135 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.449940 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.449988 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.458398 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.462190 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.472472 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.473275 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.482849 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.509039 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.516873 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.527173 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:24.527894 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.528431 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.530786 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.533517 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.547524 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.548885 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.551268 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.565595 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.572789 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:24.574828 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.598997 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:24.644138 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:24.648715 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:24.680442 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.700695 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:24.710222 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.731616 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.732905 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.735967 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.745447 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.746928 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.747368 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.747891 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.748597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.753335 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:24.761763 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:24.774245 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.780803 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.794456 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:24.808424 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:24.816867 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.817782 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.819442 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.824353 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.857889 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.874598 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.876606 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.888707 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.889151 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.892791 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:24.894657 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:24.938675 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:24.960882 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:24.968433 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:24.969904 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:24.971249 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:24.971362 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:24.972060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:24.979135 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:24.988847 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:24.998446 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:02:25.013027 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:25.038226 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.040467 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.045195 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.048029 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.059648 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:25.078653 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:25.079700 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:25.106520 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:25.112591 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.114254 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.115757 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.116094 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.117887 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.129555 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:25.149057 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:25.174892 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:25.203234 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:25.220405 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.223109 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.227000 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.237006 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.249003 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.251556 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.257193 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.290403 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:25.292145 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:25.293695 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.296262 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.302956 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.310216 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.310809 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.316918 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:25.326891 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:25.330953 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.354023 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.356626 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.369258 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.440114 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.458884 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.469975 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.478565 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.486611 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.486995 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.498147 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.519778 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.522137 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.536368 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.536686 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.540915 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.541162 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.543470 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:25.557762 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:02:25.559301 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.568190 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.568464 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.569159 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.573359 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.577236 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.580803 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.585305 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.588011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.603790 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:25.718734 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:25.732892 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:25.733361 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:25.743795 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.748355 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.751089 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.755987 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.758108 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:25.759608 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:25.769740 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:25.770334 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.780678 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.781424 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.794142 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.799486 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:25.815778 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.856005 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:25.862918 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:25.864114 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:25.876880 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:25.878875 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:25.886482 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:25.892074 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:26.006822 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.014367 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.014999 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.015584 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.015816 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.018095 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.041118 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.044145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.055362 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.060488 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.061260 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.076404 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.082551 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.085708 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.087597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.098006 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.110742 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.112682 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.113560 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.120232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.120813 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.129734 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.131306 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.144240 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:26.160446 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:26.165583 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:26.169423 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:26.260438 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:26.261919 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:26.266819 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.274022 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.287009 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.298034 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.320480 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.344995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.350148 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.361268 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.366706 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.370424 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.371121 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.372370 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.377431 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.381368 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.386924 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.393970 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.430933 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.448002 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.456130 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.465644 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.469089 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.471632 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.477164 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.502419 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.508378 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.513082 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.515830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.517678 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.518704 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:26.559355 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:26.563657 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:02:26.564648 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:26.570713 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:26.594555 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:26.622257 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:02:26.645211 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.652745 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.653014 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.654495 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.664088 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:26.665874 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:26.666339 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.667026 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.667163 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.685360 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.700140 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.713095 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:26.736919 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:26.737462 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.751735 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:26.756322 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:02:26.757553 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:26.769672 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.771258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.774757 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.777097 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.787338 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.798815 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.804931 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.828403 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.840765 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.846767 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.847702 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:26.873425 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:26.892822 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.897055 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:26.898811 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.899101 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.914304 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.916410 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.979602 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:26.985491 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:26.989218 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:26.992439 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:26.994509 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:26.995340 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:26.996768 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:02:26.997483 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:26.997565 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:27.012926 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:27.050130 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:27.072167 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.072897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.091424 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.091442 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.099989 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.114931 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:27.139489 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:27.154836 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:27.176056 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.177395 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.179069 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.181639 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.184484 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.209801 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.216039 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.217516 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.217614 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.230998 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.242767 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:27.251233 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:02:27.252177 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:27.266223 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.275527 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.284410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.286266 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.292554 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.294575 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.301179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:27.372994 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:27.403770 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:27.407706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:27.408421 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:27.409471 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.413144 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.422416 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.424793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.428772 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.438086 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.438299 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:27.450266 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:27.454082 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:27.461122 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.470458 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.478305 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.478578 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:27.494942 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:27.505203 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.505956 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.512794 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.524540 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:27.527597 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:27.539494 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:27.562296 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.576459 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.583895 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.584085 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.599464 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.611802 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.615716 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:27.622988 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:27.625384 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.709842 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.710192 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.715093 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.722266 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.722546 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.732403 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.739238 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.740340 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.741007 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.754298 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.764365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.772944 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:27.786289 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:27.813692 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:27.848335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.856408 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.867163 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.873082 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.875216 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.881120 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:27.881321 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.889086 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:27.890421 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.892324 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:27.907611 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:27.910604 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.936331 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:27.950452 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:02:27.963788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:27.967609 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:27.986208 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:27.989897 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:27.995523 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.008728 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.080509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:28.090013 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:28.093464 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.100526 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.100783 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:28.117712 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:02:28.121964 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:28.125432 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.150673 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.152587 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:28.153366 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:28.156659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.165884 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.172513 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.174733 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.184299 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.210353 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.231928 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:28.233373 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:28.233492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.239863 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.244558 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.254424 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.255389 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.260159 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.260674 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.276423 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.292016 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.292197 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:28.334141 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:28.367695 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.371152 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.384491 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.388917 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.392199 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.396489 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:28.437170 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:28.458747 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.462227 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.462536 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.467278 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.469322 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.482378 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.497332 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.498843 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.513312 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.516870 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.519249 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.521508 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:28.548985 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:28.557596 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:28.575556 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:28.576370 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.587092 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.593050 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.608084 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.619298 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.683224 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.691373 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.692987 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.697486 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.701124 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.705689 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.707132 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.712149 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.715054 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.747951 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.758900 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.759524 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:28.811262 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:28.856711 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.856711 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:28.857394 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.860085 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.860788 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.870667 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:28.876339 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.881109 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.901561 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.902146 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.906157 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:28.906231 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:28.924232 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:02:28.952558 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:28.986322 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:28.986971 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:28.988263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:28.996459 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.010882 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:29.028786 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.043105 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.075819 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.080235 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.085640 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.099447 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.100322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:29.104889 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:29.116545 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:29.121696 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.122418 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.125162 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.129693 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.134874 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:29.143504 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.146310 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.154179 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.225821 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.235389 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:29.238292 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:29.240020 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.246037 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:29.246210 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.264991 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.266350 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.285319 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.285318 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.287175 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.315073 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.317936 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.348361 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.360981 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.366269 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.366807 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.367802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.376844 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.376993 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.379345 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.381238 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:29.384486 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.402387 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.406142 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.415628 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:29.427136 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:29.428828 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:29.475085 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.519567 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.519567 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.521932 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.525776 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.534255 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.552496 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:29.552980 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:29.592765 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.609650 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.612131 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.623302 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.638778 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.642690 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.644874 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.655045 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.659178 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.663923 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.667393 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.667431 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.692144 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.696578 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.723479 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.725693 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.726205 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.751895 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.754726 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:29.770844 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:29.779083 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:02:29.790742 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:29.802302 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:29.819502 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.819525 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.820819 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.825060 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:29.831022 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:29.834708 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.846096 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.856204 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:29.864564 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.869020 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.869891 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.874486 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.877316 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.953137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:29.958123 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:29.968818 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:29.971168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:29.971374 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:29.990860 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:29.993868 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:30.001595 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:30.005753 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.025111 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:30.028990 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.034274 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.034543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.037400 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.037894 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.043694 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.044261 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.063341 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.078850 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.090345 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:30.091768 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.096422 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.097608 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.163271 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.168323 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.180575 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:30.186108 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:30.193540 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.194805 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.220600 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.227128 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.236006 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.253609 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.253638 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.254878 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.260857 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:30.287406 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:30.303133 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.309732 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.318260 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.320923 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.321995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.344468 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.347081 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:30.354434 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.355589 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.355671 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.358321 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.360659 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.375375 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:30.385155 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.398824 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:30.437130 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:30.448182 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.460440 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.462769 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.464738 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.480732 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:30.483399 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:30.495844 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.509189 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.514855 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.514902 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.515316 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.515731 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.540732 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.614404 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.617523 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.617625 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.627947 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.628808 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.629171 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:30.654733 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:30.671260 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:02:30.698186 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.704028 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.704326 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.706049 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.739002 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.757328 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.773221 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.775847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.783912 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.792058 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.793069 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.799822 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.801169 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:30.810173 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.810695 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.819096 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.822705 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.824366 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:30.852989 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:30.874150 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:30.891158 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:30.906248 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:30.910856 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:30.916183 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:30.924892 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:30.940062 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:30.951606 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.024894 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.025938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.043565 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.059000 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.064759 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.066141 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.070714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.072858 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.074426 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.083884 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.095084 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.096782 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:31.099359 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.102402 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.108441 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.109637 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.117398 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.168226 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.176643 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.177296 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.196898 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.202116 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.204047 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.208418 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.210122 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:31.246434 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:31.280701 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.297405 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.299932 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.300151 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.302191 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.302223 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:31.310225 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:31.310971 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.360077 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.361891 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.363236 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:31.365962 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:31.368425 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.369216 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.404835 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.415066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.418388 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.420125 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.423502 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.430624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.437244 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:31.485045 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.532877 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.536499 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.539686 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.563638 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.564558 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:31.565376 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.569053 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.570909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.597436 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.601747 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:31.603256 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:02:31.608115 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:31.620548 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.621618 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.642632 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.661881 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:31.670898 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.679095 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.685567 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:31.692597 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:31.696419 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.697114 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.700572 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.707224 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.710840 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:31.719695 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:02:31.720603 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.732921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.737115 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.745231 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:31.754364 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:31.808725 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:02:31.841509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.860734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.865864 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.866858 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.872115 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:31.872524 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.876981 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.888419 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.888921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.894203 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:31.895525 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.903218 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:31.907847 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:31.915925 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.917908 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:31.954015 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:31.955086 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:31.955330 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:31.979285 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:31.981102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:31.991883 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:32.021837 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.025321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.029004 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.035233 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.037201 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.044936 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:32.105134 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:32.120450 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.124213 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.129509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:32.145670 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.152890 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.159376 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:32.161856 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:32.162712 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:32.163508 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:32.180902 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:32.181098 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:02:32.181995 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.188387 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.188748 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.196343 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.202001 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.212503 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.219116 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.222844 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.234226 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.235543 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.235726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.235782 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:32.309888 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:32.316021 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:02:32.369558 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.372210 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.377113 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.386212 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.397243 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.412806 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.415877 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.422242 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.422839 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.431722 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.445416 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.451678 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:32.456593 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:32.457587 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.463526 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.475006 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.475333 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.481297 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.485386 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.487296 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.494340 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.495355 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.501765 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.503273 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:32.507433 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:32.514483 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.637392 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.641788 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.649509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.671129 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.677675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.683127 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.687505 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.688297 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:32.694548 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.701541 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.702343 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:32.704423 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:32.708894 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:32.712661 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.715847 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:32.720036 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.721630 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.724112 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.726524 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.734217 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.756664 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.768500 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:32.848757 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:32.878705 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.899407 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.909415 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.912832 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.923615 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.925674 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.938529 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:32.938706 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.945494 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:32.947926 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:32.951587 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.953167 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:32.959426 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:32.965122 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:02:32.972988 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:02:32.976635 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:32.998576 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:32.998642 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:32.998934 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.000854 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.003427 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.012199 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:33.031491 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:33.076164 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.082749 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.085235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.090202 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.091698 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:33.107326 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.107769 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:33.112217 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:33.180073 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.184789 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.186522 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.191725 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.205348 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.214066 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.223795 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.252296 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.263043 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.268538 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.272759 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.275202 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.277786 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.283874 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.300514 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.312259 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.320737 6486 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:33.320868 6486 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:33.321341 6486 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:33.321861 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:33.322001 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:02:33.322443 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:33.322834 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:33.322908 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:33.323195 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.323397 6486 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:33.330338 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.420068 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.422616 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.425565 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.425976 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.428072 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.434542 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.442710 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:33.450520 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.452425 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.461158 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:33.472115 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.480238 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.487179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.514195 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.515513 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.534679 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:33.536015 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:33.539710 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.542848 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.545645 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.546293 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.552992 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.555163 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.564816 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.566601 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.581980 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.583631 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:33.653321 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:33.670058 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:33.721045 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:33.723440 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.724439 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.737859 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.755988 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.757078 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:33.759310 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.773136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.774456 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.781450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.782024 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.794565 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.795673 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.798825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.802038 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:33.804948 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:33.806167 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.818866 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.826967 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:33.834491 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:33.841307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:33.846216 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.848889 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:33.849771 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.854406 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:33.854408 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:33.886056 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:33.894730 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:02:33.928079 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:34.005663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.010735 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:34.010910 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:34.013254 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.013749 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.014513 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.019527 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.028869 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.035974 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.047123 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.051313 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.066201 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.075224 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:34.107211 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:34.137486 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:34.162652 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.163192 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.168473 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.171030 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.171249 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.200903 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.259824 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.274991 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.275380 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.278465 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.300084 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.302608 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.304370 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.305922 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.306059 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.319135 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.322633 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.331017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.338685 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.349313 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.349959 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.350145 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.351068 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.369014 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:34.381748 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:02:34.399649 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:34.409327 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:34.493027 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:34.495486 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.506975 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.512336 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:34.514829 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:34.520921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.524369 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.542649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.543581 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.544939 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:34.545554 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:34.550004 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.567461 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.586361 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.587484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.603870 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:34.606267 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:34.608649 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.611917 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.616052 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.630720 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.654870 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.654870 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.663762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.669155 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.669519 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.672463 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:34.680922 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:34.697484 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.719537 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.728744 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.731601 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.732314 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.743969 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.745972 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:34.769037 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:34.833828 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:34.858343 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.864248 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:34.869467 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:34.879951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.899921 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:34.901419 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.923735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:34.928864 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:34.934957 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:34.945145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.946261 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:34.948400 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:34.953071 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:34.958748 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:34.959033 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:34.961660 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:34.967105 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:34.983819 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:34.985834 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.002166 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.003193 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.004873 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.017041 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.026921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.030957 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.035732 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.044732 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.050910 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:35.097846 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:35.141659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.145277 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.159080 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.166688 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.175591 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.180496 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.182228 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.187697 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.188443 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.192951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.205176 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.211014 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:35.273828 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:35.288529 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.288966 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.293694 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.304363 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.307690 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.328359 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:35.334573 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:35.357725 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:35.360138 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.372556 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.382172 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.382596 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.385320 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:35.391383 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:35.396462 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:35.396800 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:02:35.400081 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:35.400417 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:35.402077 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:35.410387 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:35.423280 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:02:35.428355 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.430143 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.438825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.442106 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.458788 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.495951 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.496554 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:35.501432 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:35.516000 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.516634 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.519930 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.535050 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.537729 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.547508 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.549520 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.550894 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.563840 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.565447 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.693035 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.693706 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.693917 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.699218 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.719487 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.726809 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.727638 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.737731 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.737802 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.741585 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.749545 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.768761 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.769176 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.769807 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.779016 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.796342 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.796633 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.800747 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:35.812552 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:35.874768 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.881453 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.881807 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.884286 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.893079 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:35.913383 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:35.919586 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:35.953392 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:35.970448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.973019 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:35.975658 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:35.978318 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:35.984447 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:35.991055 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:35.992987 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:35.996313 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.006203 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.006794 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.011034 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.014199 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.151113 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:36.152482 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.152645 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.153720 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.170665 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:36.177026 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.185847 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:36.207254 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.207541 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.225592 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.232995 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.243542 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:36.245014 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.245934 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.250941 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.260133 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.261816 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.264169 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.264405 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.269151 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.274451 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:36.276628 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.279596 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.287019 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.303398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.317204 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.319129 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.325529 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:36.338093 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:36.339129 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.388201 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.395116 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.396579 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:36.405197 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:36.408399 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.413381 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.427871 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.505396 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.508309 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:36.509315 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:36.512895 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.521317 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.521975 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.524159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.533550 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.533581 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.543711 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.556491 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.565124 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.570431 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:36.632186 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 66 similar messages]
I20240502 14:02:36.638649 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.639950 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.654341 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.661957 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.663867 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.667876 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.671386 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.672447 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.686689 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.687000 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.691365 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.712080 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:36.727864 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:02:36.728610 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.763764 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:36.765961 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:36.768314 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.773681 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.782253 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.782841 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:36.796806 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:36.804731 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:36.841595 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:36.848883 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:02:36.851493 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:36.898159 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.899873 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.907397 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.911175 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.913225 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:36.924535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.930399 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.940924 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.957723 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:36.958205 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:36.969734 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:36.984773 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:36.987032 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:36.989610 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.002666 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.005128 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.007339 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.009994 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:37.048677 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.101847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.114586 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.116407 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.117919 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.118224 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.134128 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.134821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.139806 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:37.152582 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.170912 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.171757 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.180596 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.196535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:37.215806 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:37.222105 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:02:37.223660 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.227756 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.235200 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.237462 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:37.249522 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.265300 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.268110 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.268509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:37.290056 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:37.302450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.304980 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.305771 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.310178 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:37.317550 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:37.321630 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:37.328192 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:02:37.346732 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.348716 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.349457 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.356045 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.362365 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.365550 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.366175 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:37.426540 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.430580 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.433069 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.438599 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.439102 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.440599 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.449422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:37.452637 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:37.460014 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:02:37.463699 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:37.532966 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.533797 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.536352 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.546020 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.556533 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.556542 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:37.565377 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:37.580307 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.585763 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.596313 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.604167 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.606896 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.613667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:37.632344 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:02:37.655079 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:02:37.720336 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:37.729705 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.735649 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.736105 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.743338 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.756064 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.757134 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:37.780650 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:02:37.792719 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:37.826217 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.832897 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.833333 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.850626 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.851600 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.859041 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.872123 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.874406 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.875245 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.890110 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.900247 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.917192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.922119 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.929195 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.932853 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:37.944422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.949697 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:37.952205 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:37.960198 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:37.961484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.972131 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:37.982295 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:37.988962 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:37.992792 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:38.004973 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:38.012382 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:38.161816 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.166903 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:38.180744 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:38.190212 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.191777 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.209473 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.221009 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:38.224927 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:02:38.232461 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.250345 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.257833 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.259655 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.259898 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.275549 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.305711 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:38.310149 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:38.314546 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.315589 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.317013 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.321648 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.337139 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.377470 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.386463 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.388938 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.391528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.392648 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:38.400544 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:38.405320 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:38.446635 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:38.454716 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.468168 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.473078 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.488806 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.489675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.497517 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.503453 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.505765 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.525636 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.545491 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.545789 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.553360 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.553612 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.555058 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.560083 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:38.570044 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:38.571473 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.574002 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.610745 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:38.629957 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:38.682924 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:38.691220 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.697249 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.699239 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.703756 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.706128 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.713462 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.715793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.716166 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.719167 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.720594 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.733776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.754581 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:38.783676 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:38.787954 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:38.800925 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:38.809487 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:38.813046 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.814862 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:38.820469 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.831188 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.839334 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.926935 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:38.931353 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:38.935101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:38.936573 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:38.937362 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:38.940728 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:39.006018 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:39.012398 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.018007 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.024425 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.025553 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.027040 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.040038 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.046329 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.054987 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:39.055272 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.056067 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.056632 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.057216 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.080719 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.094753 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.097584 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.099736 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.123478 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.123718 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.139086 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.149063 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:39.151836 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:39.162223 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.162273 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.167455 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.171834 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.180655 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.264694 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:39.264716 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.266048 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.267342 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.289078 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.299273 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.300812 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.334782 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.342443 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.345672 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.353790 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.360410 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.361191 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:39.361990 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.365970 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.369642 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:39.372859 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:39.375459 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.382174 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.395603 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.409099 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.487936 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.489414 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.490597 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.490661 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.492328 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.504951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:39.518507 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:39.527431 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:39.543383 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.543720 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.551715 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.557070 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.557708 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.581748 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:39.600991 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:39.635778 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:02:39.658818 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.663386 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.671258 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.686038 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.697117 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.710363 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.711439 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.719281 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.730901 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.733295 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:39.733688 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.737522 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:39.737998 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.740978 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:39.753872 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:39.776988 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:39.798067 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:39.839813 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.853447 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.853818 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:39.864859 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:39.874663 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.879755 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.880894 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.890440 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:39.890439 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.891273 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:39.895783 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.902931 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.912369 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.927469 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:39.929461 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:39.929634 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:39.931558 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:39.955510 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:39.961131 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:39.962561 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:02:39.974874 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:39.987534 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:40.021468 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:40.033820 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.034147 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.042383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.050021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.066123 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.072459 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:40.082670 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:40.137023 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.141719 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.143131 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.143571 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:40.147976 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:40.155089 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.159243 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.159943 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.176249 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.177417 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.183967 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.189549 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.205859 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:40.228826 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:40.260581 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:40.265060 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.280903 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.281395 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.281991 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.285259 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.303545 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.304627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.310561 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.312534 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.315598 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.332801 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.344182 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:40.399600 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:40.428829 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.447474 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.455865 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.461511 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.465449 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.475551 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.476445 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.480921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.484023 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.501531 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.509621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.520717 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.530328 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.532110 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.537523 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.548910 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:40.557816 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:40.567301 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.568007 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:40.588984 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:40.660857 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.677654 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.679751 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.682286 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.682858 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.692270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.733683 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.735031 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.746698 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.756613 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.757000 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.782277 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.783876 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.787746 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:40.791082 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:02:40.795367 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.800666 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.812951 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.820992 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:40.851076 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:40.852001 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:40.852963 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:40.856046 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:40.860759 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:40.880657 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.886247 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:40.893548 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:40.944023 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:40.965415 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:40.966854 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:40.969301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.977066 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.978099 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:40.987283 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:40.989297 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.990517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:40.991732 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:40.992928 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:40.996510 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.008224 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.021015 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.030323 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.033357 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.055999 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.067286 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.075163 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.077929 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.093858 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.108150 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.180698 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.182078 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.186340 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.202507 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.202661 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.222393 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.234156 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.237375 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.244746 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.251189 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.253504 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.266460 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.322037 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.323278 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:41.328372 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:41.334658 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.345176 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.345784 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.345957 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.425956 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.426723 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.428754 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:41.431401 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:41.444139 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.444892 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.463321 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.463989 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:41.466674 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:41.467123 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:41.485271 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:41.487563 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.490656 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:41.503577 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:41.508522 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:41.508522 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:41.513298 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.519280 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.526424 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.540412 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.540630 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.541225 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.563606 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.568375 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.668792 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.672492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.694259 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.699505 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.716423 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.723475 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.726251 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.729614 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:41.736389 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 66 similar messages]
I20240502 14:02:41.740036 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:41.742332 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:41.776219 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.780337 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:41.780907 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:02:41.784582 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.790712 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.802964 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.808248 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:41.817052 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.822037 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.826761 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.828871 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.842352 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.846580 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.864921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.868499 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:41.876916 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:02:41.893581 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.916731 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:41.927362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:41.931866 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:41.936889 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:41.939754 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:41.959556 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:42.009003 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:42.017086 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:42.023837 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.029840 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.030737 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:42.034204 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.036265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:42.040958 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:42.051155 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.059563 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:42.072166 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.099406 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.100744 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.100824 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.110450 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.120083 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.127907 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.135351 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.139153 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.142235 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.155458 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.156358 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.163336 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:42.197818 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:42.237615 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.267582 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:42.275812 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.278535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.286962 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.292415 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.295398 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.299517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.314637 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.315342 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.319402 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.324812 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.325282 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:42.329763 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:02:42.331084 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:42.336083 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.344240 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.345639 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.347453 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.351557 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.352659 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:42.353514 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:42.367630 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:42.368113 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:42.382208 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:42.416649 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:02:42.424844 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:42.495028 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:42.501271 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.504146 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.511276 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.512820 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.521659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.524487 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.526845 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:42.542057 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:02:42.556954 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.566159 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.568444 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.577248 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:42.583652 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.588311 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.592159 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.624486 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:42.633447 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:42.650344 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.664005 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.664920 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.669278 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.681882 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.684216 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.687006 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.699051 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.713739 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.720662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.723575 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.730973 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.734220 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.738178 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.746124 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.754827 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.780864 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.828223 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.831102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:42.833549 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.843451 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.857518 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.879968 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.885650 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.889686 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:42.892248 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.903287 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.904084 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.914906 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:42.954419 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:42.972802 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:42.978065 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:42.979101 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:42.983214 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:42.992813 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:42.997390 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:43.003363 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:43.010934 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:43.045686 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:43.087571 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.104568 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.107335 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.111467 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.111480 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.121830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.130300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.131531 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.153290 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.154124 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.154778 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.175932 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.208401 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.226999 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.227419 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.232496 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.236888 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.247164 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.252933 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.257537 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.273468 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.274294 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.275658 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:43.290853 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:02:43.303674 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:43.342214 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:43.365483 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.369537 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.370131 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.377997 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.379951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.401433 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:43.411651 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:43.445966 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:43.473940 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.474682 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.476958 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.478611 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:43.487686 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:43.495858 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.497490 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.499089 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:43.503851 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:43.508080 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.523733 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.536984 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.555001 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.561466 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:43.578475 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:43.584328 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.591614 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.599499 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.601383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.625721 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.640488 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.698041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.698843 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.712806 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.727154 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.735296 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.749805 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.760990 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.786847 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:43.791172 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.797572 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.806267 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.806565 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.813261 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.824992 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.849871 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.852922 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.854373 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.860222 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:43.861846 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:43.916553 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:43.916849 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:43.917174 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:43.933706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:43.941185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:43.947662 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:43.971920 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:44.031703 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.036873 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.054558 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.054626 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:44.064879 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:44.067198 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.067600 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.070819 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.078711 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.109143 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.109436 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.112994 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.118191 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.118481 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.122788 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.124701 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.127112 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.128331 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.128449 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:44.130666 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:44.161942 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.163033 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:44.185928 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.190629 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.191885 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.217859 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.219336 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.219892 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:44.231005 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.239967 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.344429 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.352699 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.358428 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.359268 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.359305 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.368242 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.369554 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.386391 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.386793 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.387317 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.406574 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.409409 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.409664 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.417001 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.432163 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.432173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.446146 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.458200 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:44.469038 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.490658 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:44.509510 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:44.541358 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.555193 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.558010 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.561204 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.566848 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.574132 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.650808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.653046 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.657047 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.658480 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.664124 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.664489 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:44.666970 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:44.667753 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.674468 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.682417 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.684484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.691670 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.720065 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.724890 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.724946 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.733990 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.745851 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.754753 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:44.755128 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.756245 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.768669 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.772866 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:44.796065 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:44.796203 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.835381 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.856032 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:44.864463 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:44.865272 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.872480 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.880149 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.880478 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.882761 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.916078 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.924132 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:44.927822 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:44.940586 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.941082 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:44.941704 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.946681 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:44.971943 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:44.975598 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:44.983577 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:44.994269 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:44.996927 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:44.996948 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:44.999436 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.009209 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.009469 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:45.026772 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:02:45.033551 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:02:45.076627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.080101 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.090430 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:45.097327 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:45.098932 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:45.100446 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.101255 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.124151 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.137746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.145748 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.146884 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.147406 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.153082 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.172603 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:45.183964 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:45.266157 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.278990 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.288090 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.291121 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.304944 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.306550 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.306963 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.312502 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.313604 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.317590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.332422 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.344672 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.389964 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.390201 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.395988 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.412238 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.417301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.418013 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.425787 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.432338 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.439407 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.440968 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.445302 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.456459 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.483933 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.485445 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.487227 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.491438 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.496081 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.518180 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:45.536324 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:45.555733 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:45.661018 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:45.662432 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.667008 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.669765 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.673676 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.682351 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.684372 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.692104 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.702206 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.706027 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:45.706315 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:02:45.714916 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.717511 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.722184 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.725709 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.736465 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.741231 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.750504 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.758692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.768851 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:45.824549 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:45.829820 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:45.830066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.832952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.834784 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.859910 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:45.896036 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:45.899107 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:45.965144 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:45.970176 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:45.972851 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:45.974722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.979849 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:45.980237 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:45.992041 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.000180 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.003571 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.014498 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.014808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.016415 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.021826 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.024120 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.025399 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.034061 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.043148 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.067883 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:46.072747 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:46.079581 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:46.082491 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:46.128716 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:46.129117 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.142653 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.143462 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.153788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.156140 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.161506 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:46.170251 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:46.263985 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.268962 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.274469 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.290985 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.292434 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.293207 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.293418 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.306393 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.307729 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.312163 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.314517 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.317428 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.317472 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.317646 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.322911 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.323092 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.364524 14859 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:46.364634 14859 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:46.365034 14859 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:46.365341 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:46.365483 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:02:46.366577 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:46.366672 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:46.366705 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:46.366842 14859 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:02:46.380585 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:46.401772 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.403658 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.508984 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.511145 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.520391 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.521469 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:46.521503 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:46.527398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.528095 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:46.536582 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:46.552949 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:46.609390 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:46.610853 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:46.624923 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.626435 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:46.632463 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:46.647747 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.647854 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.659618 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.671610 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.671739 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.675707 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.687898 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.690403 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.690410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:46.771327 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:02:46.791512 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.792994 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.801427 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:46.816049 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 94 similar messages]
I20240502 14:02:46.820220 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.837298 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.840929 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.844434 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.844853 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.853467 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.860841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:46.867110 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.889322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.908608 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:46.928138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:46.945248 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:46.946934 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:46.949210 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:46.955150 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:46.959689 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:46.997269 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.005105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:47.015547 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:47.019568 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:47.024052 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:02:47.027631 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.036746 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.040632 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.043399 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.125481 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:47.126371 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:47.135545 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.139338 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.148726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:47.160346 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.164999 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.166759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.169162 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.169930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.172016 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:47.179208 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:47.182370 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.185281 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.188143 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.195555 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.206457 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:47.206631 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.220942 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.227293 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.236732 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:47.243995 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.261642 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:47.264005 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.266675 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.267597 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.268393 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.268940 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:47.286688 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:47.287425 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.306273 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:47.319825 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:47.356590 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.391950 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.404945 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:47.412402 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.413244 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.431370 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:47.439447 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:02:47.451665 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.451787 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:47.454735 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.464646 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.465444 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.474584 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.478092 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.486845 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.490298 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.490417 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.499565 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:47.506770 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:47.508452 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.519827 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.520509 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.530864 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.531522 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.539990 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:47.540362 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:47.543687 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.554555 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.557885 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:47.562431 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:02:47.562726 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:47.580602 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:02:47.582839 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:47.611672 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:47.657349 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:47.732448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.761898 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.765147 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.767407 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.775360 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.776116 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.777165 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.785439 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.792145 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.798874 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.806542 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.806720 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.857066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.868566 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.871001 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.871636 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.872601 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.877032 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.883353 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.888185 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:47.889549 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.902344 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:47.918083 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:47.931505 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:47.992915 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:47.997924 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:47.999868 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.001565 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.011180 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.025027 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:48.045615 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:48.070407 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.071897 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.072790 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.073601 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.083940 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.098012 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:48.118237 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:48.158106 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.159798 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.161875 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.168008 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:48.168531 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.169270 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.173041 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:48.210315 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:48.243326 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:02:48.258877 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.265467 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.274103 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.280995 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.283648 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.292342 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.333931 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.334292 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.335297 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.344703 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.347591 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.350909 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.354759 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.361315 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.367856 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.368821 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.396765 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.412562 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.456800 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.464383 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:48.466181 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:48.467157 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.483471 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.495091 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.513851 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.513887 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:48.520854 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.539366 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.544225 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.547151 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.549846 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:48.556859 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:48.571619 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.613016 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.620999 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.626591 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.627653 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.633208 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.634733 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:48.677264 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:48.719339 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.741572 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.741887 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.745118 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.747180 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.747654 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.755966 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:48.765780 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.769284 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.772507 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.774127 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.784248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:48.792642 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:48.792765 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.793558 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.794584 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.799607 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.813884 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.823304 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.840777 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:48.851711 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.900017 17337 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:48.900141 17337 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:48.900763 17337 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:48.901039 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:48.901206 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:02:48.903120 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:48.903224 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:48.903263 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:48.903646 17337 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:48.918170 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:48.927745 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.933483 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:48.934891 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.935225 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:48.965628 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.969473 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:48.973753 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:48.976109 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:48.989044 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.991158 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:48.991158 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:48.994163 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:48.997822 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.048929 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.051501 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.053164 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.055107 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.064424 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.085835 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:49.092386 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:49.121924 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.122385 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.127568 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.129204 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.129515 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.137763 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:49.149266 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:49.230029 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:49.237303 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.255640 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.259439 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.259963 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.264353 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.269290 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.270772 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.271448 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.278105 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.306962 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.316105 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.317747 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:49.351540 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.388597 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.389190 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.394748 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.398840 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.400713 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.403389 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.411646 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.416741 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.417380 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.431450 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.456316 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.459187 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.514997 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:49.521451 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:49.528378 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.533283 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.543053 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.544745 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.546494 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:49.589423 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:49.596117 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.634778 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.649454 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.649725 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:49.657752 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:49.658922 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.660847 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:49.678857 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:49.682269 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:49.683420 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.684566 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.691257 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.706286 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.709116 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.721654 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.722121 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.731855 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.734794 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.735242 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:49.742362 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:49.753979 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.754078 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.757503 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.758301 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.761219 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.761855 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.769309 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.782197 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.808949 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:49.810004 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.817047 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:49.827574 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:49.847976 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:49.891193 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.914675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.914786 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:49.915370 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.933434 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.934043 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:49.937579 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.939071 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:49.962381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:49.962965 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:49.967432 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:49.976661 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:49.977072 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:49.982252 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:49.986972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:50.042218 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:50.079813 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.092442 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.093986 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:50.104343 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:50.104408 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:50.109189 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:50.127846 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.128631 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.129293 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.216218 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.219748 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.227260 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.230311 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.230989 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.232852 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:50.248772 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:50.250151 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:50.256505 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:02:50.261937 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.264477 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.269277 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.270957 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.271601 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.289537 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.292682 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.308492 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.312438 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.316653 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.324468 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.325902 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.332163 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.353391 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.366143 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.374078 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.375452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.491309 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.495498 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:50.500470 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:02:50.504626 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.514469 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.514907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.519979 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.520450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.523567 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.568729 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.569577 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.579473 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.596396 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:50.601869 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:50.603927 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.606062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.627135 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.630379 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.636132 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.642339 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.646198 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.657467 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:50.660681 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:50.668910 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.670158 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.681838 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.687573 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.689642 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.696375 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.697597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.698596 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.712260 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.717849 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:50.737254 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:02:50.751292 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:50.825488 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.832485 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.841279 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:50.849350 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:50.856707 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:50.857797 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.863540 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:50.872689 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.880453 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:50.888417 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:50.894068 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.895792 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:50.903604 18435 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:50.903718 18435 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:50.904171 18435 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:02:50.905306 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:02:50.905432 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:02:50.906008 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:50.906105 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:50.906141 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:50.906349 18435 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:50.923828 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.929157 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:50.982919 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:50.985635 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.006825 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.019204 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.019331 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.020606 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:51.041498 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:51.071287 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.071848 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.088470 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.096540 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.098399 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.110505 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.117599 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.121922 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.126682 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.127463 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.132912 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.136947 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:51.153048 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:51.172552 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:51.223857 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.224136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.232003 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.233680 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.234541 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.237952 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.309976 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:51.312446 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:51.320501 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.320892 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.330731 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.332942 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.336427 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.345216 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.347055 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.375597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.381057 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.406780 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:51.428028 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:51.430771 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.431181 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.442782 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.452085 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.454793 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.456357 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.460631 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.465696 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.499541 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.501331 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.513322 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.515101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.517946 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.552345 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.565951 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.569363 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.570816 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.573377 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.579384 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:51.600314 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:51.654119 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:51.657876 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:51.669661 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.671017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.671226 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.675467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:51.697001 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:51.702520 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:51.703852 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.704453 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:51.706866 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:51.707923 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.709199 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.716434 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.733929 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.736719 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:51.820493 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:02:51.824631 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 14:02:51.839335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:51.856792 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:51.857369 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.858429 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.858830 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.872159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.873948 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.874630 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.875764 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.879786 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.902172 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.907328 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:51.916725 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.951388 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:51.956593 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:51.957228 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:51.957509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:51.969163 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:51.971191 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:51.982059 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.035595 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.036370 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.039575 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:52.047744 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:02:52.047986 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:52.048538 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:52.061304 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.061789 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.155162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.157058 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.157939 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.174553 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.174726 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.177332 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.178611 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.187978 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.194145 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.197942 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.205986 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.206158 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.210798 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:52.214934 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:52.221436 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:52.230182 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:52.238711 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:52.257256 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:52.260095 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.273692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.274421 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.288491 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:52.295461 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.318358 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.319437 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.319954 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:52.321380 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.328445 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.334892 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.338826 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:52.353029 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:02:52.391538 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.395071 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.399030 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.406841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.407544 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.408774 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.418467 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:52.421077 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:52.445703 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.489670 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.497750 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.510641 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.510712 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.515790 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:52.518731 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:02:52.519685 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.527678 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.537712 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.544724 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.546212 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:52.546497 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.550984 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:52.553517 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:52.563805 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.567199 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.567432 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:52.591949 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:52.638345 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:52.641084 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.648741 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.652089 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.663849 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.665534 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.674597 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:52.689631 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:02:52.692705 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:52.727653 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.732651 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.737215 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.752457 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.765919 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.771162 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.798555 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.803166 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.822438 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.824208 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.828123 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.833377 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:52.873242 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:52.889086 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:52.901443 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:52.903769 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.906919 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:52.915684 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:52.921350 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.951288 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:52.959586 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:52.983093 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:52.995870 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:52.999267 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.011442 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.026067 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.028750 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.033852 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.044068 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.059371 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.073905 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.075259 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.083092 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.084970 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.085918 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.086766 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.098635 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.131700 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.176944 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:53.187577 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:53.190716 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.191085 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.192067 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.203182 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.224612 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.236582 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.240402 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:53.257597 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:02:53.258440 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.266405 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.273397 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.274191 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.295365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:53.300061 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:53.305311 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.307128 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.323567 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.325467 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.326726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.393082 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.396960 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.399276 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.410373 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.411511 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.420162 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:53.464869 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:53.465579 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.472496 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.472815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:53.473655 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:53.485600 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.498020 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.510784 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.516441 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.520526 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.531363 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.531955 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.532094 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.532240 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:53.568537 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:53.624284 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:53.645418 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.650616 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.650890 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.660326 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.673020 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.683393 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.711104 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.717898 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.724186 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.739553 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.743366 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.752066 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.769440 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.772655 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.780558 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.783779 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:53.788172 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.809291 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:53.824735 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:53.825171 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:53.853286 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.863688 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:53.864954 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.864996 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:53.867067 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:53.867301 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:53.883957 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:53.920962 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:53.966142 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:53.968443 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:53.969623 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:53.989881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.001356 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.002104 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.004489 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.006736 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.017923 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.020515 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:54.029225 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.035655 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.058091 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.072727 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.076674 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.079387 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.079576 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.087733 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.088594 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.099049 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.101403 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.116461 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.117050 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.128657 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.144405 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:54.201311 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:54.271147 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.276594 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.283780 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.294987 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.303189 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:54.306142 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.322422 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.325044 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.340947 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.342680 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.343123 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.344127 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.345167 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.353957 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.363409 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.372506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:54.373450 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:54.381146 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.382953 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.395193 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:54.403081 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.406934 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.428354 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.431756 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.434305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.435968 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.450132 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:54.498768 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:54.522837 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:54.547919 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.554579 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.556905 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.579007 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.586987 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.590834 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.594529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.608023 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.615393 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.619578 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.644505 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.649983 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.654620 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.655763 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.665176 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.683907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.690712 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:54.697326 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.706064 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.711217 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.722477 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:54.725512 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.726713 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.731524 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.740574 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:54.741971 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:54.758661 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:54.819684 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.827697 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.828078 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.830871 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.833309 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.845685 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:54.846378 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.852831 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:54.853509 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:54.870393 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.885421 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.889281 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:54.889653 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.893704 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:54.897691 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.901166 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:54.911830 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:54.913102 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:54.956296 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:54.976483 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:54.976855 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:54.980684 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:54.981315 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:54.987362 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:54.989122 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:54.989619 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:54.990525 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:55.055416 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:55.087354 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:55.090627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.096436 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:55.104074 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:55.107398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.121631 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.128499 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.134145 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.141085 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.141423 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.144726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.147467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.167819 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.179994 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.195895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.198700 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.199191 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.204672 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.216742 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.217366 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:55.227551 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:55.232967 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:55.254637 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:02:55.282027 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:55.333566 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.341295 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.344889 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.345180 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.357141 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.357398 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.410663 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.429688 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.439904 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.440537 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.442693 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.444634 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.445837 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.454653 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.458379 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.467263 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.469641 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.503899 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.504341 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.504580 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.514154 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.515616 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.518230 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.533195 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.535890 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.550489 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.552361 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.570854 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.572235 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.572758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:55.621479 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:02:55.692880 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:02:55.737461 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.738942 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.748732 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.755755 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.761745 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.765995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.781348 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.807868 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.810509 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.818221 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.820621 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.826195 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.828205 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.832098 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.832942 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:55.837244 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:55.846639 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.847925 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:55.850574 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.854952 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:55.864470 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:55.870731 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:55.879914 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.883546 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.889039 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.898298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:55.898478 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:55.904677 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:55.917167 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:55.932240 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.939899 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:55.942440 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:55.946043 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:55.947899 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:56.037531 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.043041 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.044529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.052345 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.053517 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.066076 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.129428 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.129685 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.131014 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.133170 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.143411 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.144205 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.166394 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.166399 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.177249 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.177592 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.177824 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:56.179132 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:56.181757 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:56.260114 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:56.294457 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.294909 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.299831 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.301352 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.303716 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:56.320219 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:56.320844 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.327251 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.333001 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.343240 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:56.343596 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:56.345713 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:56.352563 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.355654 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.357398 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:56.389464 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:56.405711 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.412184 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.415343 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.426246 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.428087 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.442121 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.519129 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.526465 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.527578 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.531682 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.532002 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.538281 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.544456 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.559427 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.565289 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.565918 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.583705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.596522 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.649343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.651751 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.654999 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:56.658064 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:56.658892 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.673702 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.674873 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.756891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.759260 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.783846 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.783888 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.793785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.797169 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.807773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.810106 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.812114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.816255 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:56.825750 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 14:02:56.840852 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:56.841378 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:56.849164 21841 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:02:56.849290 21841 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:02:56.849789 21841 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:02:56.851059 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:02:56.851944 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:56.853164 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:02:56.853273 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:56.853324 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:02:56.853484 21841 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:02:56.853680 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:02:56.853816 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:02:56.860002 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.871750 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.879142 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.885771 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:56.898610 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:56.901641 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:56.903072 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.905896 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:56.909758 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:56.928993 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.929582 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:56.936755 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:56.936825 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:56.956444 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:56.957830 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.037165 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:57.065066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.070142 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.077445 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.078425 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.088862 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.093307 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:57.097018 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:57.103900 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.105865 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.109802 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:57.118299 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.120393 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:57.121160 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.122546 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.141162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.144961 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.148454 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.151118 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.157737 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.165026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.201153 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:57.225035 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:57.226704 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:02:57.229853 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.237160 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.253691 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.313794 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:02:57.324978 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.332942 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:57.333256 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.335909 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.338435 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.348035 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.350684 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.351487 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.352124 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.356590 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.368382 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.382426 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.385298 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.387410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.394814 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.395781 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.400101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.415442 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:57.424399 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:57.446130 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.448778 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.452538 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.459523 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:57.470352 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.479053 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:57.489735 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.494603 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.495688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.500757 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:57.518467 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:57.519585 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.525094 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.537855 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.538102 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.538839 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.550271 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:57.563057 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.624941 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.644776 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:02:57.650115 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:02:57.670518 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.684979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.687943 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.688550 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.692629 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.711649 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.716624 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.716647 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.722903 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.724525 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.738590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.739987 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.740149 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.740329 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.751524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.768889 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:57.770364 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:57.772143 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:57.807016 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:57.808578 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:57.844920 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:02:57.897708 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:57.913434 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:57.922204 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.929710 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.943063 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.944864 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:57.953492 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:57.955634 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.961603 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:57.963073 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:57.976811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:57.991848 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.007270 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.017012 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.018947 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.019686 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.037552 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.037957 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.038669 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:58.137461 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:02:58.160192 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.164885 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.165266 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.189898 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.193182 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.193677 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.193935 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.195089 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.201020 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.201756 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.208449 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.213902 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.224241 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.240554 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.260892 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.273231 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.276930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.277575 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:58.324816 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:58.343668 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.348024 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.348906 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.357209 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.358484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.372314 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:58.401890 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:02:58.437135 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:58.449112 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:58.455487 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.456825 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.467694 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.467895 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.472164 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.478309 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.487030 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.499122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.501207 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:58.504474 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:58.504644 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.511574 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.569388 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.571535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.573170 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.576370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.578224 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:58.609655 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:58.611883 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.671977 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.677512 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.685897 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.703544 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.705327 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.707638 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.710057 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.714133 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:58.734450 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:58.734628 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:58.739967 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.740554 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.761055 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:58.803994 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:58.804349 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:58.804450 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.816179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.818570 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.819715 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.822602 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.826172 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:58.837917 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:58.840553 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.845094 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.846457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:58.867056 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:58.874195 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:58.878834 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:58.975453 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:58.987943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:58.989580 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:58.995216 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.004356 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.004706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.009230 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.018847 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.020004 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.020326 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.021732 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.036866 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.052489 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.095561 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.096027 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.104874 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.108523 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.113234 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.120707 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:59.200253 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.208611 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.209766 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.212936 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.234974 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.247426 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.255789 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.257650 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.259972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.272023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.288708 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.291245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.297305 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.322517 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.341779 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.341851 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:59.348623 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:59.361554 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.365211 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.388159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.388763 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:59.399715 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.400133 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.401294 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.405623 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.409898 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.418354 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:02:59.422775 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:59.434229 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:59.501104 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.508333 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.522979 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.528458 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.528545 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.537719 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:02:59.551640 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:02:59.559975 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:02:59.565714 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.585156 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:59.585455 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.601655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.611608 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.615912 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.622774 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.624470 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.633112 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.636435 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.639461 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.645113 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.648188 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.660691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.677371 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:02:59.689191 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.692420 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.693986 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.708305 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.708774 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.713649 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:59.742980 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.745142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.757447 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.789865 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.791313 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.792784 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.798437 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.800709 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.801028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.845471 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:59.849048 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:02:59.869453 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:02:59.875262 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.877085 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:02:59.886729 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:02:59.887117 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:02:59.889436 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:02:59.902150 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.905390 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:02:59.923710 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:02:59.927619 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:02:59.955662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:02:59.957679 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:02:59.958992 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:02:59.960557 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:02:59.962334 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:02:59.963375 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:02:59.981808 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:00.018352 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:03:00.029481 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.037776 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.043156 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.055848 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.057597 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.073088 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:00.075362 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:03:00.102367 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:03:00.116200 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.125573 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.125898 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.145258 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.148252 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.151219 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.165362 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.166383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.171856 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.189551 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.194260 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.203629 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.217458 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.218926 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.243274 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.244666 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.246344 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.254717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.365849 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:00.369118 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:03:00.369134 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.379076 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.380028 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.387357 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.388437 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:00.390576 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:00.392323 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:00.394174 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:00.395268 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.397332 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.433346 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.435959 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.437690 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.440328 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.442552 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.451445 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.457777 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.465637 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.484794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.568766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.574594 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.577236 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.586447 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.605206 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.607734 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.629666 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.667243 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.672410 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.674801 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.687065 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.695698 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.697378 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.697503 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.709424 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.715071 21841 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:00.715168 21841 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:00.715533 21841 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:03:00.717160 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:00.717372 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:00.717473 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:00.717504 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:00.717710 21841 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:00.725678 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:03:00.725821 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:03:00.735107 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:00.737231 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:03:00.738245 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:00.740494 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.750803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.760448 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.765838 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:00.770511 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:03:00.772429 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.784878 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.818475 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:00.823464 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:00.870133 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.877497 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:00.878720 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:00.884958 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:00.899497 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.902989 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:00.945660 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:00.978031 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:00.978533 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.979125 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:00.981354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:00.982935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:00.991369 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:01.007431 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.020406 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.022291 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.024590 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.037551 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.049914 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.050992 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.057276 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.062211 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.067867 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.093071 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.103592 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.111385 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.151873 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.157577 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.157866 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.164990 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.166689 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.175033 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:01.186486 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.243929 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.247912 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.250757 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.267927 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.268996 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.276697 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.305887 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.307526 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.310694 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:01.317438 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:01.323484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:01.332795 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.338657 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.344404 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.369096 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.370344 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.371706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.374630 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.384552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.418123 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.476315 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:01.487308 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:01.491515 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.498183 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.503082 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.503269 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.508466 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.518826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.525018 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.525678 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.547257 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.547588 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.566665 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:01.568851 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:01.606235 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.615350 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:01.625003 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.626190 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.629251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.637048 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:01.662976 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.673583 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.674280 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.675175 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.675206 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.676136 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:01.676450 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.685112 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.695722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.788661 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.792075 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.798048 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:01.801568 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.803339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:01.809463 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:01.829135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.831223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:01.833039 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:01.836674 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 14:03:01.840577 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.842211 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.855346 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:01.867873 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:01.881939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:01.943076 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:01.951454 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:01.963392 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:01.964031 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:01.964903 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:03:01.965502 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:03:01.968222 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.972450 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:01.992158 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:01.997259 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.012017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.013178 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.017469 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.020421 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.025120 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.025816 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.026196 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.029045 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.036536 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.040454 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.070894 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.085800 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:02.086164 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:02.172230 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:02.194769 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.202857 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.204452 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.206743 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.209913 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.221058 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.221459 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:02.222530 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.230068 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.248420 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:02.250252 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.258145 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:02.270653 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.274423 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.276664 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:02.287276 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:03:02.287942 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.290175 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.305310 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.305620 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.307735 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.310834 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.323582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:02.326630 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.339558 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.341368 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.348114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.355738 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:02.358994 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:03:02.359948 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:02.360229 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.364332 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.392879 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:02.412447 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:02.447074 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:03:02.471292 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:02.479862 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.495330 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:02.497217 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:02.500118 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.503880 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.522384 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.526451 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.529331 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.544348 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.548385 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.564836 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.570346 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.575115 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.587054 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.587194 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.599547 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.600992 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.605538 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.622220 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:02.652896 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:03:02.668982 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:03:02.683658 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.686669 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.688131 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.692410 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.710201 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:02.723623 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:02.743845 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.840150 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.841040 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.842214 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:02.848307 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:02.848714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:02.856493 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:03:02.860153 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.860469 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:02.862856 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:02.862946 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.863233 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.864653 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.875715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.879037 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.886289 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.898180 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:02.912349 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.921073 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:02.928120 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:02.928217 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.932323 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.939589 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:02.940536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:02.941773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.945504 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:02.968082 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:02.976783 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:02.978606 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:03.022413 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:03:03.113602 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.119933 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.132200 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.137652 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.140662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.146198 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.147764 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.148809 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.160421 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:03.169983 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:03.171540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.174342 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.180912 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.182375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.188431 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.191764 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.204869 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.207883 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.215888 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:03.230542 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:03:03.292625 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:03.322207 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.331192 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.331420 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.333014 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.346959 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.370369 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.398388 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.400034 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.403746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.425118 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.428184 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.445869 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:03.451296 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:03.463773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.467954 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.468703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.474654 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.475685 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.486377 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.546631 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.546711 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.550531 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.551167 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.557195 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.588920 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.608548 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.626156 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.636744 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.638702 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.640000 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.650082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:03.712971 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:03.717994 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:03.727881 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.744215 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.746574 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.747759 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.747782 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.761313 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.771881 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.774178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:03.780892 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:03.784783 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.787864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.804301 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.818464 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:03.852249 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:03.877924 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.885944 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.892992 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:03.907151 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.909503 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.922825 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.941181 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:03.944716 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.947101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.951566 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:03.956532 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.957595 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:03.970609 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:03.994776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:03.994824 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:03.995080 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:03.999312 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.004302 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.019178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:04.028000 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:04.101666 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:04.111572 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.115824 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.123464 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.127869 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.132390 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:04.133697 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.162961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:04.185714 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.185791 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.194021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.194707 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.196734 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.197279 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.206043 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:04.230746 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.256265 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.261212 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.261540 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.262181 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.263082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.280913 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.299006 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.305249 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.313482 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.335039 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.336207 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.340090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.376595 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.385032 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.394217 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.394217 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.395255 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.404754 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:04.414423 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.476135 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.481586 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:04.493228 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:04.496449 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.497416 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.498363 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:04.502169 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.504336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.507032 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.512372 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.515455 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.523339 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:04.526707 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.532059 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.545106 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:04.563303 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:04.568571 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:04.602650 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:04.605118 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:04.608975 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.616490 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:04.617934 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.619062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.637115 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:04.679472 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:03:04.694669 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.694763 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:04.696501 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.699720 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.701471 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.701529 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.703758 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:04.709686 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.714062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.772643 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.775357 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.775738 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:04.785712 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:04.786909 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.792862 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.807090 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:04.871122 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.891433 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:04.899124 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:04.920009 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:04.926326 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:04.935139 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.949867 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.951500 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:04.961727 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:04.970763 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.979375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.984418 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:04.986375 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:04.990638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:04.995940 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.005481 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.006129 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:05.011389 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:05.017163 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.026984 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.029217 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.032564 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:05.035331 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:05.036672 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:03:05.089541 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:05.097625 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.098238 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.100638 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.110533 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.113234 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.130439 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.157622 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.169581 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.169992 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.182783 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.191877 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.215324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.256546 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.262542 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.267807 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.268332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.276973 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.300621 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.316762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.317023 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.322270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.324383 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.328677 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.345369 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:05.354558 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:05.415776 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.420879 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.430032 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.435681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:05.440505 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:05.441601 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.453517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:05.474068 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:05.495478 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.500130 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.500458 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.504830 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:05.514322 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:05.515395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.518920 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.529306 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.537846 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.542943 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.548458 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.558856 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.561206 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:05.587249 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:05.621982 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.628051 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.635881 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.636538 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.639470 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.649859 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.716200 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.724383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:05.727679 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:03:05.731014 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.734982 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.740788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.741607 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.755242 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.783676 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.785945 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.786729 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.799522 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.800235 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.864041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.869493 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:05.874863 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:05.875097 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.878942 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.897431 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:05.917935 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:05.931690 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:05.933770 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:05.937654 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:05.957158 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:03:05.959144 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:05.965823 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:05.975900 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:06.000331 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:06.016129 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.019870 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.032047 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.037590 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:06.040715 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:06.040992 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.044986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.083989 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.089167 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.115388 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.116653 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.121876 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.123741 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.125850 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.126305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.144899 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.148528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.152204 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.160457 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:06.192870 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:03:06.212062 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.235885 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.237752 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.240667 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.254925 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.256318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.267822 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.281140 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.285382 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:06.291847 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:06.296241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.304958 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.305537 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.309576 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:06.316210 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.393651 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.399399 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.410750 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.417160 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.421042 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:06.421334 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:06.435130 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.445412 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.453011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.454653 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.459987 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.473227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.474839 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:06.505414 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:06.565726 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.572713 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.576763 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.577100 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.594550 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.598517 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.604835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.620782 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.625017 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.633649 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.638305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.642928 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.726277 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:06.734352 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.734778 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.734808 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.740865 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.746824 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:06.752606 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.763933 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:06.767128 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.768373 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.782482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.788442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.798686 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.809165 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.813314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:06.854334 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:06.875856 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.877692 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:06.880605 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:03:06.885352 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.885844 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:06.896441 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:06.897864 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:06.898828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.918967 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:06.926250 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:06.941870 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:06.943387 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:06.951450 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:06.953572 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:06.970305 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:06.981585 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:03:07.006090 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.028553 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.037384 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:07.040895 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 14:03:07.040954 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.041258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:07.042999 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.043522 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:07.058604 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.065587 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.067198 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.067785 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.069439 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.069459 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.088454 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:07.105141 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:07.115917 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:07.127043 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:03:07.142153 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.176021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.179525 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.180503 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.185061 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:07.197652 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.216084 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.227241 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.246284 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.266037 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.281083 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:07.293581 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:03:07.295917 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.295954 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.302415 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:07.317631 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:07.322619 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:07.336675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.340874 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.351347 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.358700 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.364332 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.368943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:07.377944 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.382114 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.382777 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.403978 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.411397 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.415769 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.416325 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.462494 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.474040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:07.475298 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.482728 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.487392 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.491361 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:07.518615 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:07.534683 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:07.535106 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:03:07.564714 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:07.565114 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.566336 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:07.575081 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.576314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.588150 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.588912 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.627866 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:07.658375 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:07.691721 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.697844 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.697947 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.703150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.720166 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.737941 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:07.782516 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:03:07.797384 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.807844 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:07.814738 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.825099 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.828101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.840601 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:07.843972 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.848985 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:07.861652 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:07.864974 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:07.866354 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.870318 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:07.870803 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:07.874442 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:03:07.972028 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:07.986047 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:07.994060 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:07.998008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.002436 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.006191 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.020694 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.027698 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.028365 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:08.031195 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:08.035451 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.045158 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.053475 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.080367 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.118530 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.122079 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.129554 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.134155 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.145726 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.148362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:08.155686 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:08.211992 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:03:08.229806 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.234300 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.241854 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.257745 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.269033 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.276899 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.281283 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.292423 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.301890 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.315961 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.321389 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.322507 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.327278 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.327713 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.343544 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.349128 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.350791 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.377349 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:08.416648 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:08.424937 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:08.438403 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.445394 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.449919 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.453881 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.456138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.457965 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.479926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.487376 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:08.488127 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:08.491454 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.492177 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.492852 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.510718 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:08.537998 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:08.554306 28873 maintenance_manager.cc:419] P db037a839b724e0babeaf39df2515fc8: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033398
W20240502 14:03:08.591408 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:08.601389 28872 maintenance_manager.cc:643] P db037a839b724e0babeaf39df2515fc8: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.046s user 0.022s sys 0.000s Metrics: {"bytes_written":20351,"cfile_init":1,"dirs.queue_time_us":1510,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1950,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1064,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":19984}
I20240502 14:03:08.606528 28873 maintenance_manager.cc:419] P db037a839b724e0babeaf39df2515fc8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7001 bytes on disk
I20240502 14:03:08.607386 28872 maintenance_manager.cc:643] P db037a839b724e0babeaf39df2515fc8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 14:03:08.618041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.622463 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.636695 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.637688 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.691251 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.700878 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.701964 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.708864 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.717756 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.721521 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:08.723378 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:08.738420 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.740422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.817749 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.836422 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.838006 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.849033 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.853788 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.860482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:08.891918 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:08.893066 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.893287 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.905267 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:08.906845 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:08.920770 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.931128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:08.933945 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:08.933962 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:08.962369 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:08.971096 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:08.974010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:08.983085 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:08.988253 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.018487 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.025992 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.034543 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.035323 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.038297 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.045071 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.056504 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:09.114053 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.114786 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.118129 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.129086 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.129118 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.139699 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.157101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.178038 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.185065 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.191859 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.199294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:09.202754 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.210791 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.211061 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.264791 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.266609 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.269590 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.276320 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.278285 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.282717 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.291368 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.315183 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.315409 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.317390 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:09.322227 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:09.333840 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.358422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:09.359951 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:09.382508 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.423229 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.428301 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.433276 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:09.434260 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:09.437399 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.438841 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.456079 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.456859 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.457715 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.464452 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.464502 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.476294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:09.480741 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:03:09.486599 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:09.489125 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.490839 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:09.511947 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.543104 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.548933 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.551913 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.553059 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.554190 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.556093 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:09.601200 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:09.623081 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.658159 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.678471 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.682039 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.682955 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.683197 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:09.688367 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 14:03:09.689285 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:09.706353 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:09.727584 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:09.732465 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.735317 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.737668 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.746649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.747886 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.766790 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.774571 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.777901 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:09.784029 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.784260 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:09.788556 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.799121 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:09.799872 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:09.800532 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.806795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:09.813925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:09.819803 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:09.835125 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:09.853153 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:09.865052 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:09.914754 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:09.964680 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.004809 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.011433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.024966 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.025894 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.034914 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.037074 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.040686 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.046473 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.047510 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.047572 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.048164 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:10.072315 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.072432 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.074525 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.077162 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.085088 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.085151 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.092260 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:10.126051 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:10.128607 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.133596 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.140966 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.145771 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.150940 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.152912 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.159708 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.220614 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.220940 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:10.222012 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.223410 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.225699 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.230212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:10.232690 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.247685 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.304587 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.304590 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:10.304872 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:10.309711 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.311120 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.315088 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.317026 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.364310 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.366108 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.369280 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:10.372059 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:10.372570 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.387812 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.391795 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:10.400537 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.478475 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.486544 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.495045 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.496616 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.497684 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.500416 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.501211 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.502856 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.516451 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.542465 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.543284 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.549455 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:10.614513 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:10.643991 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.658202 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.668334 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.668715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.708238 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.719847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.731518 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.732161 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.734339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.736446 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.737537 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.740669 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.760882 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:10.763322 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 14:03:10.764524 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.769057 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:10.772169 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:10.772425 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.777621 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.780987 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.786926 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:10.801456 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:10.805030 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:10.810387 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.810745 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.814851 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.820358 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:10.841967 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.893901 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:10.900645 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:10.910364 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:10.911545 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:10.920387 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:10.924021 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.014808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.016649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:11.017058 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:11.031131 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.036543 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.036885 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.047266 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:11.098940 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:03:11.118595 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.120728 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.126842 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.132253 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:11.142184 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:11.142314 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.145722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.155112 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.168825 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:11.170320 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:11.175429 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.189718 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.199826 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.202656 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.203094 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:11.212410 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:11.228869 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.229588 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.231076 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.247620 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.248239 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.347204 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.357719 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.361727 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.362623 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.366852 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.373191 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.390866 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.401211 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.408435 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.421388 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:11.421756 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:11.429586 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.447010 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.464798 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.465886 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.468696 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.468746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.481288 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.502612 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.578595 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.587087 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.587617 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.597039 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.601048 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.603134 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.624763 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:11.632480 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:11.636013 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.642692 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:11.644812 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:11.645004 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.659668 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.675071 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:11.678145 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:03:11.713338 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:11.719101 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.720106 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.732544 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.733140 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.737717 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.740984 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.745443 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.754616 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:11.757525 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.758314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.763105 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:11.775686 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:11.794376 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.829486 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:11.837993 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:11.839329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.842638 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:11.870718 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.882125 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:11.886790 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:11.939230 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:03:11.940182 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:11.976651 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.980935 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:11.987697 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:11.999053 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.000968 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.006342 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:12.010129 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.015244 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.017536 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.017866 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.023192 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.027514 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.048884 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:12.050529 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.053972 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.060096 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.060899 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.072058 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 14:03:12.080610 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.082293 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.085883 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.090399 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.102942 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.173358 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.175105 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.176585 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.191479 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.191946 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.205448 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.232977 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.239239 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.242516 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.255600 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.259452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.263757 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:12.270529 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.271359 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.271600 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.271739 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.285360 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.285624 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.289816 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:12.299458 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:12.314232 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.318629 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.328927 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:03:12.330034 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:03:12.330421 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:12.355939 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:12.356487 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:12.447780 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.460561 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.463918 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.465315 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.475263 30760 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:12.475394 30760 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:12.475867 30760 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:03:12.476264 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:03:12.476462 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:03:12.481662 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:12.481864 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:12.481923 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:12.482228 31634 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:12.484982 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.492725 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.495239 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:12.495824 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.503247 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.515633 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.519906 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.523921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.525944 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.529265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.529289 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.532429 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.540866 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.541057 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.551173 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.556321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.556416 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.562052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.574863 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.579449 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.588624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.608975 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.713732 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:12.742851 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.747526 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.750094 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.755604 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.764683 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.781421 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.785876 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.787381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.793820 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.793834 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:12.801573 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 14:03:12.804955 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:03:12.806700 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.811658 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.882243 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.884869 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.889987 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.897367 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.912993 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.915174 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:12.917564 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:12.925032 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:03:12.925595 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:12.932443 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:12.944062 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.952721 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.953222 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:12.963222 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:12.966698 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.975870 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:12.984256 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:12.985541 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.996887 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:12.999997 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.003580 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.005743 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:13.009794 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:13.102738 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.106640 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.107100 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.111493 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.112722 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.117326 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.118324 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.127492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.135995 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.154085 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.154541 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.156667 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:13.173910 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:13.186434 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:13.213994 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.216289 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.217181 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.217715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.233336 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:13.245950 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 14:03:13.251526 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:13.260004 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:13.285631 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:13.314294 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.321768 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.324105 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.327939 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.335243 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.351277 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.385372 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.386426 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.389096 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.390985 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.408418 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.411860 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.425091 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.430467 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.432978 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.447423 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.459100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.465811 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:13.532388 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:13.568290 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.571281 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.575783 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.576768 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.581028 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(372e9cc5aaed4b80b3b451efdb3e94fc): perf score=0.033337
W20240502 14:03:13.592990 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:13.600075 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.605957 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.606922 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.621960 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.622579 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.634454 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.634522 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.666028 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.667116 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff): perf score=0.033354
W20240502 14:03:13.671355 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:13.677213 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:13.680361 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.682034 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.682570 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.687309 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.703186 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:13.704815 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.708563 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:13.718343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:13.734427 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:13.735829 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(372e9cc5aaed4b80b3b451efdb3e94fc): perf score=0.033377
I20240502 14:03:13.746969 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.757324 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(d9e0261898a445949f3cf21d2efb599f): perf score=0.033386
I20240502 14:03:13.786651 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:13.807400 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:13.821463 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(372e9cc5aaed4b80b3b451efdb3e94fc) complete. Timing: real 0.238s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":59342,"dirs.queue_time_us":28016,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1273,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":219,"threads_started":3,"wal-append.queue_time_us":4939}
I20240502 14:03:13.823114 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff): perf score=0.033403
I20240502 14:03:13.848397 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.181s user 0.032s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":40716,"dirs.queue_time_us":12447,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":3620,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12605,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":204,"threads_started":3,"wal-append.queue_time_us":18723}
I20240502 14:03:13.853621 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff): 4587 bytes on disk
I20240502 14:03:13.854382 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 14:03:13.917220 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.160s user 0.029s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":19690,"dirs.queue_time_us":6202,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":3189,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":922,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":219,"threads_started":3,"wal-append.queue_time_us":11840}
I20240502 14:03:13.918866 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f): 4840 bytes on disk
I20240502 14:03:13.919953 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 14:03:13.912779 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(372e9cc5aaed4b80b3b451efdb3e94fc) complete. Timing: real 0.177s user 0.031s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":73064,"dirs.queue_time_us":14023,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":900,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2019,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":227,"threads_started":3,"wal-append.queue_time_us":19511}
I20240502 14:03:13.928705 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(d9e0261898a445949f3cf21d2efb599f): perf score=0.033429
I20240502 14:03:13.967952 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.145s user 0.013s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":97,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":16468}
I20240502 14:03:13.969413 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(372e9cc5aaed4b80b3b451efdb3e94fc): 4925 bytes on disk
I20240502 14:03:13.970125 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(372e9cc5aaed4b80b3b451efdb3e94fc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 14:03:13.970912 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff): 4587 bytes on disk
I20240502 14:03:13.971285 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 14:03:13.995941 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.064s user 0.021s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":791,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7312,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":115,"threads_started":1,"wal-append.queue_time_us":5123}
I20240502 14:03:13.996937 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(e6787832a36648aca61dd26a363089d5): perf score=0.033334
W20240502 14:03:14.008541 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:14.049382 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:14.062315 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(e6787832a36648aca61dd26a363089d5) complete. Timing: real 0.063s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":138,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":2351,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":13684}
I20240502 14:03:14.063371 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(372e9cc5aaed4b80b3b451efdb3e94fc): 4925 bytes on disk
I20240502 14:03:14.064081 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(372e9cc5aaed4b80b3b451efdb3e94fc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.065128 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f): 4840 bytes on disk
I20240502 14:03:14.065565 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.066340 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5): 4840 bytes on disk
I20240502 14:03:14.066705 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.106443 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(e6787832a36648aca61dd26a363089d5): perf score=0.033361
I20240502 14:03:14.117837 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.134608 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.135990 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.144657 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.147729 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(e6787832a36648aca61dd26a363089d5) complete. Timing: real 0.041s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":857,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":565,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":768,"thread_start_us":102,"threads_started":1,"wal-append.queue_time_us":14359}
I20240502 14:03:14.150264 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5): 4840 bytes on disk
I20240502 14:03:14.150696 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.152513 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.171732 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(cfd44eeeec934413a5001673430e6500): perf score=0.033383
I20240502 14:03:14.187685 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.206527 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(cfd44eeeec934413a5001673430e6500) complete. Timing: real 0.035s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":983,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":616}
I20240502 14:03:14.208372 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500): 4924 bytes on disk
I20240502 14:03:14.208774 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.212805 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(ff65ce0568514e89894ec845fbda31f3): perf score=0.033394
I20240502 14:03:14.217706 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.219614 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.224853 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(ff65ce0568514e89894ec845fbda31f3): perf score=0.033394
I20240502 14:03:14.239130 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.026s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":324,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":127}
I20240502 14:03:14.241163 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3): 4586 bytes on disk
I20240502 14:03:14.241514 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.250762 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.253628 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.255508 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:14.262338 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.266759 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.277534 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.280433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.281863 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.306294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.307766 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.314215 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.346833 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.354525 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.130s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":771,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1920,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":93230}
I20240502 14:03:14.355443 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3): 4586 bytes on disk
I20240502 14:03:14.355849 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.373864 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:14.395736 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.405689 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.433012 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.446895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:14.451287 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:14.462291 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.462863 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.471375 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.475603 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.491735 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:14.498893 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.503142 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.507655 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.511813 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.513263 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.522006 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:14.533299 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:14.549985 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.568962 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(db68071f838a4e209f3db69bea6e54f3): perf score=0.033368
I20240502 14:03:14.598591 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(db68071f838a4e209f3db69bea6e54f3) complete. Timing: real 0.029s user 0.011s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3069,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1601,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":972,"threads_started":2,"wal-append.queue_time_us":4565}
I20240502 14:03:14.599588 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(db68071f838a4e209f3db69bea6e54f3): 4586 bytes on disk
I20240502 14:03:14.600010 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(db68071f838a4e209f3db69bea6e54f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.607165 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(db68071f838a4e209f3db69bea6e54f3): perf score=0.033382
I20240502 14:03:14.623138 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(db68071f838a4e209f3db69bea6e54f3) complete. Timing: real 0.016s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1675,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":562}
I20240502 14:03:14.624118 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(db68071f838a4e209f3db69bea6e54f3): 4586 bytes on disk
I20240502 14:03:14.624637 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(db68071f838a4e209f3db69bea6e54f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.632797 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:14.634068 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.639492 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.644510 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.650349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.654289 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.656826 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(4aa944cd0b82429da02bd3dd35e0cce7): perf score=0.033396
I20240502 14:03:14.657147 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.669479 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.673409 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.675628 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.678689 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.695706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:14.696655 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:14.714851 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:14.719266 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.728935 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.735776 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.737493 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.742784 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.752763 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.767760 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.778275 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(4aa944cd0b82429da02bd3dd35e0cce7) complete. Timing: real 0.121s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":105435,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1570,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":196,"threads_started":2,"wal-append.queue_time_us":100058}
I20240502 14:03:14.779209 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(86cfc7d5709f414aac098ae3c05a48a3): perf score=0.033428
W20240502 14:03:14.779958 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.811628 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:14.820325 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(86cfc7d5709f414aac098ae3c05a48a3) complete. Timing: real 0.041s user 0.012s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":903,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":2778,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":24155}
I20240502 14:03:14.821799 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7): 4925 bytes on disk
I20240502 14:03:14.822232 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 14:03:14.823025 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3): 4840 bytes on disk
I20240502 14:03:14.823437 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240502 14:03:14.827395 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 14:03:14.847163 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:14.861392 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.875449 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:14.875761 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.882768 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:14.892679 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:14.925910 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:14.930135 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:14.937240 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.959379 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:14.969030 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.981014 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:14.992028 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:14.993046 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:14.996155 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:14.999111 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:15.003353 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:15.003422 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:15.009593 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.012022 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.012532 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.016474 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.022994 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.090291 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:15.096526 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.098130 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.105178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.117501 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.120785 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.121467 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.124629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:15.129966 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.137101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.138403 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.138510 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.141191 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.148541 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.246003 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(015b587864d54da8aee36c3afd968e86): perf score=0.033359
I20240502 14:03:15.248509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.252960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.261258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.291508 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.298547 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.313755 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.314818 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.315639 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.324663 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.327749 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(252f4087c0e5443093e856fe676839b5): perf score=0.033379
I20240502 14:03:15.327999 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.335346 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:15.336135 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.351189 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.354687 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(252f4087c0e5443093e856fe676839b5): perf score=0.033389
I20240502 14:03:15.366679 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.039s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":15076,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2484,"threads_started":2,"wal-append.queue_time_us":3916}
I20240502 14:03:15.367803 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5): 4925 bytes on disk
I20240502 14:03:15.368453 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.368402 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.122s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13523,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":2003,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":116,"threads_started":2,"wal-append.queue_time_us":68113}
I20240502 14:03:15.369344 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86): 4840 bytes on disk
I20240502 14:03:15.369771 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.370586 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(015b587864d54da8aee36c3afd968e86): perf score=0.033390
I20240502 14:03:15.377795 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(0669ab13586a48ff9382fe1b80f47553): perf score=0.033393
I20240502 14:03:15.387638 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3317,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":256}
I20240502 14:03:15.388640 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86): 4840 bytes on disk
I20240502 14:03:15.389775 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.393146 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.038s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1919,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":2551,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":159}
I20240502 14:03:15.394047 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(0669ab13586a48ff9382fe1b80f47553): perf score=0.033398
I20240502 14:03:15.405294 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(0669ab13586a48ff9382fe1b80f47553) complete. Timing: real 0.027s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":6787}
I20240502 14:03:15.406229 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553): 4586 bytes on disk
I20240502 14:03:15.416580 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553) complete. Timing: real 0.010s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240502 14:03:15.417357 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:15.427177 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(0669ab13586a48ff9382fe1b80f47553) complete. Timing: real 0.032s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":1000,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":81,"threads_started":1,"wal-append.queue_time_us":1041}
I20240502 14:03:15.428169 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5): 4925 bytes on disk
I20240502 14:03:15.428611 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.429610 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553): 4586 bytes on disk
I20240502 14:03:15.429930 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.472393 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.476151 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.476328 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.503631 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.504093 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:15.506752 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:15.528451 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.530942 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.540321 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.546257 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.560259 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.562453 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:15.573390 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.580870 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:15.590860 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:15.611465 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.612068 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.618613 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.621498 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:15.626806 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:15.628237 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.638846 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.639550 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.676868 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.684790 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(5a91fd318f0844c1961f16823a7ccc88): perf score=0.033343
I20240502 14:03:15.698962 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.702201 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.705009 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.706527 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.711057 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.731581 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.739856 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.744520 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:15.746074 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 14:03:15.753882 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(5a91fd318f0844c1961f16823a7ccc88) complete. Timing: real 0.069s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1259,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":472,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":49354}
I20240502 14:03:15.757030 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88): 4840 bytes on disk
I20240502 14:03:15.757436 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.761708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.775468 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.781841 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.862802 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.871816 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(f05a1a0bf1314d098f2bf65df58222e2): perf score=0.033395
I20240502 14:03:15.876624 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.884557 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:15.895674 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:15.900388 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:15.912979 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:15.919363 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.924414 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(f05a1a0bf1314d098f2bf65df58222e2) complete. Timing: real 0.052s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10541,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":2,"wal-append.queue_time_us":6570}
I20240502 14:03:15.925171 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2): 4925 bytes on disk
I20240502 14:03:15.927333 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":883,"lbm_reads_lt_1ms":4}
I20240502 14:03:15.938889 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:15.945662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:15.946918 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:15.948233 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:15.952180 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:15.955801 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:15.978416 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.029278 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.050406 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.051954 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.060158 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.060856 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.069496 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.075212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.082507 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.083335 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.095762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:16.098707 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.106331 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.109107 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:16.156545 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.185981 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.197479 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.205451 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.218767 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:16.227357 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:16.232299 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.258016 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.258141 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.266316 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(1dda4193aaaf459fb5049ff49541229a): perf score=0.033351
I20240502 14:03:16.278590 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.284523 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.284659 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.285930 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.288759 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.291399 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.296314 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.300640 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.300803 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.305665 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.306106 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.320991 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.055s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11141,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":5148,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":214,"threads_started":2,"wal-append.queue_time_us":26080}
I20240502 14:03:16.321872 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a): 4840 bytes on disk
I20240502 14:03:16.322249 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
I20240502 14:03:16.393153 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(1dda4193aaaf459fb5049ff49541229a): perf score=0.033384
W20240502 14:03:16.409817 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:03:16.414436 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":943,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":5554,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":4677}
I20240502 14:03:16.415206 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a): 4840 bytes on disk
I20240502 14:03:16.416101 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 14:03:16.421685 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(a75497cf22014c0cb9d079b448284b3d): perf score=0.033395
I20240502 14:03:16.421943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.429208 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.429396 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(4b1230a7230f48558da1069a5988b981): perf score=0.033395
I20240502 14:03:16.437502 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.442034 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:16.454362 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:16.457576 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.479146 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.485832 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(4b1230a7230f48558da1069a5988b981) complete. Timing: real 0.056s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":27895,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":27381}
I20240502 14:03:16.486640 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981): 4586 bytes on disk
I20240502 14:03:16.487334 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:03:16.495417 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(a75497cf22014c0cb9d079b448284b3d) complete. Timing: real 0.074s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34050,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":3759,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":33270}
I20240502 14:03:16.496426 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d): 4925 bytes on disk
I20240502 14:03:16.496852 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 14:03:16.513679 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.515645 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.524426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.529628 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.536878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.564452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.567174 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.568223 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.571238 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.581720 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.583156 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.584803 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.593535 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.594883 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.600183 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.619117 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.642095 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.655718 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:16.709103 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.712054 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.714749 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.717386 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.718335 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.718766 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.739200 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:16.740200 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:16.755448 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:16.811650 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.819115 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.827811 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(785828dee4594ca8813b9418cc5dba2d): perf score=0.033353
I20240502 14:03:16.845006 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.850045 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7620,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":234}
I20240502 14:03:16.853525 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d): 4925 bytes on disk
I20240502 14:03:16.854624 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.856655 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.858552 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.870985 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.015s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14747,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3}
I20240502 14:03:16.884476 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:16.889731 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:16.907153 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.908345 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.908782 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:16.916393 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.917878 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.920800 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(20e960d4e55047918a1dad82fae21ecd): perf score=0.033377
W20240502 14:03:16.925287 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:16.927568 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.927786 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:16.952525 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:16.956596 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:16.965369 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:16.971174 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:16.982623 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:16.983214 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:16.989698 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.069s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":16545,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":4304,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":669,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":30829}
I20240502 14:03:16.990703 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(0176fb1a25a94c1c9d8cf0dc63577e91): perf score=0.033396
I20240502 14:03:16.995318 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd): 4840 bytes on disk
W20240502 14:03:16.996146 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:16.997591 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 14:03:16.998409 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(785828dee4594ca8813b9418cc5dba2d): perf score=0.033397
I20240502 14:03:16.998741 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(20e960d4e55047918a1dad82fae21ecd): perf score=0.033398
W20240502 14:03:17.012324 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.015987 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.017853 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.019s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":181}
I20240502 14:03:17.019373 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d): 4925 bytes on disk
I20240502 14:03:17.020162 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240502 14:03:17.035179 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.042165 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.043s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7119,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":3755,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3048,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":4403}
I20240502 14:03:17.043545 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(0176fb1a25a94c1c9d8cf0dc63577e91): perf score=0.033409
I20240502 14:03:17.043973 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(0176fb1a25a94c1c9d8cf0dc63577e91) complete. Timing: real 0.053s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":170,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":2534,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":482,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":134,"threads_started":2,"wal-append.queue_time_us":7014}
I20240502 14:03:17.044875 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91): 4586 bytes on disk
I20240502 14:03:17.046362 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240502 14:03:17.053916 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240502 14:03:17.056829 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:17.066892 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(0176fb1a25a94c1c9d8cf0dc63577e91) complete. Timing: real 0.022s user 0.013s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2992,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1967,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":84,"threads_started":1,"wal-append.queue_time_us":562}
I20240502 14:03:17.067798 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd): 4840 bytes on disk
I20240502 14:03:17.068146 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.068822 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91): 4586 bytes on disk
I20240502 14:03:17.069123 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.114313 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.116225 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.116819 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.118479 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:17.121928 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:03:17.126039 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.128288 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.152700 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.158946 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.160296 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:17.176673 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:17.183277 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.206190 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:17.207855 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:17.215307 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.222419 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.231006 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.234429 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.263737 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:17.265611 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.266706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.268563 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.271466 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d): perf score=0.033375
W20240502 14:03:17.290282 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:17.291280 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.296885 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.297775 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d): perf score=0.033380
I20240502 14:03:17.299738 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.310746 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.316166 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1103,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":121}
I20240502 14:03:17.317137 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d): 4840 bytes on disk
I20240502 14:03:17.317525 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.320534 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(dcc536794ffb453495d025f29ea152bc): perf score=0.033386
I20240502 14:03:17.320935 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.332645 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.340168 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.019s user 0.009s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1238,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":446,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":60,"threads_started":1,"wal-append.queue_time_us":404}
I20240502 14:03:17.341008 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc): 4586 bytes on disk
I20240502 14:03:17.341387 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.343422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.350065 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.078s user 0.009s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":1555,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106,"threads_started":1,"wal-append.queue_time_us":37883}
I20240502 14:03:17.350891 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d): 4840 bytes on disk
I20240502 14:03:17.351418 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.375658 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(81b2d5d0f4044106bed079c3faa72447): perf score=0.033401
W20240502 14:03:17.401793 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.412781 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(81b2d5d0f4044106bed079c3faa72447) complete. Timing: real 0.037s user 0.016s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6245,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":9742,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":306,"threads_started":2,"wal-append.queue_time_us":8910}
I20240502 14:03:17.413694 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(dcc536794ffb453495d025f29ea152bc): perf score=0.033411
I20240502 14:03:17.429659 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.441242 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.443346 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:17.444849 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:03:17.445506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.451419 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:17.456223 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:17.472637 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.059s user 0.014s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":131,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1206,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":39410}
I20240502 14:03:17.473701 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447): 4925 bytes on disk
I20240502 14:03:17.474505 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.475339 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc): 4586 bytes on disk
I20240502 14:03:17.475682 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.477811 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.481011 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:17.488416 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:17.488621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.496101 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.498066 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.501029 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.508051 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:17.553200 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.561440 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.586103 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.605939 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.608094 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.608372 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.616212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.636435 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.636940 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.655000 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.655714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.670150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.684499 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:17.686343 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:17.710309 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:17.740782 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.741242 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.752372 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.761996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.764812 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.775256 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.789398 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:17.796341 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.800621 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.813235 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.819335 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:17.824656 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:17.827497 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:17.835760 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:03:17.836426 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:17.837085 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:17.844089 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964): perf score=0.033353
I20240502 14:03:17.853197 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling FlushMRSOp(ed713f89a4fd47d79f03711dd14c1334): perf score=0.033357
I20240502 14:03:17.860152 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:17.868023 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:17.871214 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: FlushMRSOp(ed713f89a4fd47d79f03711dd14c1334) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4029,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1238,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":250}
I20240502 14:03:17.871901 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(ed713f89a4fd47d79f03711dd14c1334): 4925 bytes on disk
I20240502 14:03:17.872391 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(ed713f89a4fd47d79f03711dd14c1334) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 14:03:17.875480 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.887131 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:17.889663 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:17.891494 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.047s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2222,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":139}
I20240502 14:03:17.892333 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964): 4840 bytes on disk
I20240502 14:03:17.901191 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 14:03:17.923575 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:17.957988 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:03:17.981637 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(19784717201a464c84607d2a43e651c2): perf score=0.033389
I20240502 14:03:18.033908 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.037694 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.043733 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(19784717201a464c84607d2a43e651c2) complete. Timing: real 0.062s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":5435,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":121,"threads_started":2,"wal-append.queue_time_us":1317}
I20240502 14:03:18.045185 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(19784717201a464c84607d2a43e651c2): 4586 bytes on disk
I20240502 14:03:18.045861 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(19784717201a464c84607d2a43e651c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 14:03:18.052534 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.053746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.056877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.063087 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.078794 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.086596 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.090925 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.097095 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.109391 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.115708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:18.116814 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:18.133450 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.136539 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:18.136710 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 14:03:18.145048 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.150947 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.160389 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.180657 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.198489 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.200821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.207173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:18.209177 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 14:03:18.212802 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:18.216164 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.220821 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.224300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.291293 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.297535 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling FlushMRSOp(c88f85f143d845a1b7066244c606f8ba): perf score=0.033337
I20240502 14:03:18.300417 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:18.301910 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:18.316069 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.317564 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: FlushMRSOp(c88f85f143d845a1b7066244c606f8ba) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":1311,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":599,"threads_started":1,"wal-append.queue_time_us":2291}
I20240502 14:03:18.318555 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba): 4840 bytes on disk
I20240502 14:03:18.318938 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 14:03:18.321981 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.322605 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling FlushMRSOp(15513e220e144883b216c4f3ac8ade0b): perf score=0.033344
I20240502 14:03:18.329073 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:18.329173 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:18.346441 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:18.365679 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.367898 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.371986 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.391570 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.392496 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.395074 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.401878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.403273 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling FlushMRSOp(2b1adebdbd104796b927963e86a15295): perf score=0.033367
I20240502 14:03:18.425701 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.427624 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.436823 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.444802 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:18.458498 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:18.458684 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.463348 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.494433 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: FlushMRSOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.172s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":87441,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":581,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":84475}
I20240502 14:03:18.497033 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b): 4586 bytes on disk
I20240502 14:03:18.497474 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 14:03:18.502976 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: FlushMRSOp(2b1adebdbd104796b927963e86a15295) complete. Timing: real 0.100s user 0.011s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12630,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":4542}
I20240502 14:03:18.503862 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(2b1adebdbd104796b927963e86a15295): 4925 bytes on disk
I20240502 14:03:18.504580 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(2b1adebdbd104796b927963e86a15295) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:03:18.548698 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.548698 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.556123 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.565713 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.583159 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.595134 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.596388 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.598753 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.598789 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.599647 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.600435 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.602105 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:18.658576 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:18.706972 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.712771 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.722898 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.728207 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.734349 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.742985 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.753922 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.754323 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.760399 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.762840 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:18.797355 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:18.821171 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:18.824914 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.830526 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.830826 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.849458 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.865432 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.866107 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.871224 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.874954 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.891908 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.896435 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:18.913287 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.915695 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:18.915776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:18.918689 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:18.979280 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:18.985316 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:18.985962 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.009469 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.010936 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.012925 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.079078 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.088469 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.090320 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.099118 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.104454 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:19.122545 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.129401 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.165475 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.172376 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.174496 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.182257 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.184478 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.223141 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:19.235581 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.245041 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:19.250007 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.257654 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.261181 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.266669 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.271232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.273041 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:19.276952 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.277622 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.285269 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.285630 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.304962 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.317355 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.317407 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:19.342624 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:19.361611 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:19.366762 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:19.391480 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:19.406488 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:19.419049 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.421628 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.422653 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.427983 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:19.432427 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.442739 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.452391 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.453894 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.456705 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.465847 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:19.481379 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.502346 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.505681 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.510036 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.516155 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.534454 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.537421 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:19.538908 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:19.547188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.548415 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.549556 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.554423 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.556991 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:19.565711 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.579205 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.579278 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.590399 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.591441 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:19.623369 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:19.636174 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 14:03:19.666512 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:19.672178 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.672470 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.676012 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.679047 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.694679 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.709059 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.742374 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.744922 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.747110 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.752581 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.753666 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.769383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:19.831431 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:19.846392 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:19.861583 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:19.881084 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.891697 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:19.893891 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.897918 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.899391 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.914695 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.936183 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:19.951807 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:19.953544 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:19.964454 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:19.964931 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:19.976357 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:20.050951 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:20.063510 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.063510 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.071828 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.075528 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.078105 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.088888 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.107107 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.122336 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.124167 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.126720 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.130427 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.131793 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:20.137243 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:20.161670 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.174314 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.193528 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.199306 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.207195 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.237751 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.246930 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.255307 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.256494 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.261090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.275528 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.280469 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:20.288480 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:20.340962 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.353282 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:20.354212 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:20.360263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.372336 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.373657 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.389041 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.395897 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:20.401230 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:20.401691 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.406975 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.413091 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.415923 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.424317 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:20.456310 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:20.526644 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.526947 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.530133 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.534337 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.540635 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.541551 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.542796 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.543455 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.549391 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.552901 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.568629 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.600952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.623026 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.623615 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.624724 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.625021 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.636287 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:20.645627 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:20.648536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:20.656558 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:03:20.685887 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:20.694399 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:20.705315 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 14:03:20.710915 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.715452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.721554 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.731644 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.734338 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:20.800706 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:20.834192 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:20.838094 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.839747 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.851011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.854086 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.855801 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.862071 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.870445 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.872793 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:20.874325 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.884956 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.887945 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.892426 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:20.896008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:20.897501 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:20.903327 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:20.926352 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:20.926846 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:20.932750 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:20.936985 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:20.955644 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.018052 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.023797 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.031800 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.033682 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.034916 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.050199 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:21.081195 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:21.089989 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.122741 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.126353 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.126585 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.134150 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.138413 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.177980 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.191217 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.191766 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.195755 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.204178 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.204609 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.205425 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.214224 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.214445 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.222616 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:21.228631 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:21.239334 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.258569 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.279143 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.320233 2658 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:21.320381 2658 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:21.320890 2658 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:03:21.321382 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:03:21.321532 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:03:21.322721 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:21.322942 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:21.322997 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:21.323122 2658 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:21.356220 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.362697 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.368170 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.371706 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.372745 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.385367 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.386634 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.399266 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:21.402035 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:21.405246 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.422870 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.430828 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.436625 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.468868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.470600 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.473950 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.474224 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.474727 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.487169 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.583141 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.586827 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.597347 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.616184 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:21.621816 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.635738 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.637578 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.671828 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.675714 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:21.682627 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.686604 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.687273 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.706996 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:21.715021 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.723555 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.726229 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:21.738056 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.739126 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.742321 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.744251 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.747246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:21.747308 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:21.747498 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.756572 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.758340 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.767323 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.771539 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:21.772513 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:21.797906 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:21.799968 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:21.808387 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:21.819849 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:21.836644 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:03:21.877024 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:21.887490 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:21.888149 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:21.888806 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:21.893082 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:21.901127 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:21.908931 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.918401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:21.938441 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:21.940089 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:21.941869 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(785828dee4594ca8813b9418cc5dba2d): perf score=0.033349
I20240502 14:03:21.992035 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.050s user 0.025s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":2482,"dirs.queue_time_us":7117,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":3923,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":217,"threads_started":3,"wal-append.queue_time_us":13431}
I20240502 14:03:21.994817 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(a75497cf22014c0cb9d079b448284b3d): perf score=0.033353
I20240502 14:03:22.031179 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(cfd44eeeec934413a5001673430e6500): perf score=0.033358
I20240502 14:03:22.032348 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.044551 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:22.045050 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:03:22.062379 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:03:22.064841 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.066210 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.067543 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.075783 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.106007 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.106595 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.112975 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.123651 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:22.139174 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:03:22.139416 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.142128 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.148272 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.156823 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.170892 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.181769 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(a75497cf22014c0cb9d079b448284b3d) complete. Timing: real 0.186s user 0.017s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2220,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":157702}
I20240502 14:03:22.182746 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(c88f85f143d845a1b7066244c606f8ba): perf score=0.033352
I20240502 14:03:22.184083 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.191965 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.219287 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.222039 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.233923 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(c88f85f143d845a1b7066244c606f8ba) complete. Timing: real 0.051s user 0.015s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1238,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":107,"threads_started":1,"wal-append.queue_time_us":13544}
I20240502 14:03:22.234836 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d): 4925 bytes on disk
I20240502 14:03:22.237314 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(785828dee4594ca8813b9418cc5dba2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.238075 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d): 4925 bytes on disk
I20240502 14:03:22.240208 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.240967 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba): 4840 bytes on disk
I20240502 14:03:22.241583 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.242287 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff): perf score=0.033361
I20240502 14:03:22.253254 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(cfd44eeeec934413a5001673430e6500) complete. Timing: real 0.222s user 0.026s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":23472,"dirs.queue_time_us":3968,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":3796,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":341,"threads_started":3,"wal-append.queue_time_us":35296}
I20240502 14:03:22.257334 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500): 4924 bytes on disk
I20240502 14:03:22.270537 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.271914 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.272789 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(dcc536794ffb453495d025f29ea152bc): perf score=0.033423
I20240502 14:03:22.278050 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.288120 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:22.291316 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.307101 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.308537 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.330984 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:22.334971 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:22.349644 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.106s user 0.016s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1303,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":46356}
I20240502 14:03:22.352989 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(015b587864d54da8aee36c3afd968e86): perf score=0.033353
I20240502 14:03:22.369948 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.096s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1346,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":548,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":16253}
I20240502 14:03:22.371027 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(f05a1a0bf1314d098f2bf65df58222e2): perf score=0.033441
I20240502 14:03:22.405058 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.052s user 0.017s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1212,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":14437}
I20240502 14:03:22.406006 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(400da91cbe234c79859b23ebe1643eb0): perf score=0.033353
W20240502 14:03:22.419358 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:22.420799 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(f05a1a0bf1314d098f2bf65df58222e2) complete. Timing: real 0.050s user 0.018s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1398,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1302,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":16058}
I20240502 14:03:22.421748 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff): perf score=0.033446
I20240502 14:03:22.447391 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(400da91cbe234c79859b23ebe1643eb0) complete. Timing: real 0.041s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":41423,"compiler_manager_pool.run_cpu_time_us":25,"compiler_manager_pool.run_wall_time_us":25,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":145,"dirs.run_wall_time_us":7157,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2789,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":7782}
I20240502 14:03:22.448472 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(ff65ce0568514e89894ec845fbda31f3): perf score=0.033361
I20240502 14:03:22.449342 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.027s user 0.015s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1380,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":3330}
I20240502 14:03:22.450623 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2): 4925 bytes on disk
I20240502 14:03:22.453706 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.454788 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(ed713f89a4fd47d79f03711dd14c1334): perf score=0.033451
I20240502 14:03:22.455698 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.455919 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.477972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.482131 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.488112 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.488976 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.039s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1381,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":118,"threads_started":1,"wal-append.queue_time_us":12296}
I20240502 14:03:22.489840 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86): 4840 bytes on disk
I20240502 14:03:22.490265 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(015b587864d54da8aee36c3afd968e86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.490931 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff): 4587 bytes on disk
I20240502 14:03:22.492244 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(ed713f89a4fd47d79f03711dd14c1334) complete. Timing: real 0.034s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1411,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":9009}
I20240502 14:03:22.492722 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.493178 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(ed713f89a4fd47d79f03711dd14c1334): 4925 bytes on disk
I20240502 14:03:22.493360 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(400da91cbe234c79859b23ebe1643eb0): 4586 bytes on disk
I20240502 14:03:22.493533 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(ed713f89a4fd47d79f03711dd14c1334) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.493660 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(400da91cbe234c79859b23ebe1643eb0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.494266 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3): 4586 bytes on disk
I20240502 14:03:22.494288 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(ff65ce0568514e89894ec845fbda31f3): perf score=0.033387
I20240502 14:03:22.494601 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240502 14:03:22.494768 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:22.507190 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.508587 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.513219 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.019s user 0.017s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":95,"threads_started":1,"wal-append.queue_time_us":173}
I20240502 14:03:22.514781 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff): 4587 bytes on disk
I20240502 14:03:22.515178 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(5e1b37b67e104154b6755c1762f8f6ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.515933 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc): 4586 bytes on disk
I20240502 14:03:22.516237 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(dcc536794ffb453495d025f29ea152bc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.516929 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3): 4586 bytes on disk
I20240502 14:03:22.517218 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(ff65ce0568514e89894ec845fbda31f3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240502 14:03:22.554875 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.570750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.572491 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.578795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.579092 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.584652 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.596899 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.602598 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.613046 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.618749 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:22.626130 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.630039 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.637604 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:22.675546 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:03:22.749307 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.750135 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.755582 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.762624 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.768627 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(86cfc7d5709f414aac098ae3c05a48a3): perf score=0.033359
I20240502 14:03:22.785274 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.787195 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:22.795996 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:22.797205 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:22.803759 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:22.805164 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.805482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.814674 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.849406 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.857612 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:22.868810 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.870405 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.875026 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(86cfc7d5709f414aac098ae3c05a48a3) complete. Timing: real 0.106s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1405,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":295,"threads_started":1,"wal-append.queue_time_us":28595}
I20240502 14:03:22.875922 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3): 4840 bytes on disk
I20240502 14:03:22.876680 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.877480 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(15513e220e144883b216c4f3ac8ade0b): perf score=0.033368
I20240502 14:03:22.878845 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.880069 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.881480 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:22.897339 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:03:22.897965 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:22.899032 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.901787 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:22.952458 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.075s user 0.018s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1307,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":928,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100,"threads_started":1,"wal-append.queue_time_us":50354}
I20240502 14:03:22.954085 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964): perf score=0.033351
W20240502 14:03:22.972157 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 14:03:22.976022 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:22.981330 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.027s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1388,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":645,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":6902}
I20240502 14:03:22.982836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:22.985031 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964): 4840 bytes on disk
I20240502 14:03:22.985770 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:22.986704 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 14:03:22.987532 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b): 4586 bytes on disk
I20240502 14:03:22.987851 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
W20240502 14:03:22.989563 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:03:22.993160 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:22.998546 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.004770 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.020136 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:23.043095 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:23.062204 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.100711 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.106261 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.110970 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:23.113178 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:23.120023 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.124419 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.130249 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.140353 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.140820 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.158896 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.166258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.168183 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.169287 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.169495 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.169839 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.185792 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.187081 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.205188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.239428 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(252f4087c0e5443093e856fe676839b5): perf score=0.033352
I20240502 14:03:23.247442 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(5a91fd318f0844c1961f16823a7ccc88): perf score=0.033388
I20240502 14:03:23.249729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.257975 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.018s user 0.013s sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":155}
I20240502 14:03:23.258975 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5): 4925 bytes on disk
I20240502 14:03:23.259352 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(252f4087c0e5443093e856fe676839b5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240502 14:03:23.263391 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:23.279245 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.288930 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.289228 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.290273 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.294669 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:23.313015 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:23.316188 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.319406 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(5a91fd318f0844c1961f16823a7ccc88) complete. Timing: real 0.071s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14585,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1825,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":552,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":108,"threads_started":2,"wal-append.queue_time_us":20908}
I20240502 14:03:23.320510 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88): 4840 bytes on disk
I20240502 14:03:23.320986 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 14:03:23.322397 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964): perf score=0.033399
I20240502 14:03:23.351889 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.029s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1297,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":136}
I20240502 14:03:23.352795 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964): 4840 bytes on disk
I20240502 14:03:23.353232 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(d5ffcf19e66e4adca28d87cbe2245964) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 14:03:23.374207 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.374953 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.377560 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.378993 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.379941 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.380573 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:23.438650 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.441385 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.445324 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.448150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.449527 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.458953 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.459192 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:23.487988 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 14:03:23.493103 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:23.564632 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.567539 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.572420 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.575832 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.579385 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.582442 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.584944 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.585004 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.604720 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(4aa944cd0b82429da02bd3dd35e0cce7): perf score=0.033392
I20240502 14:03:23.631788 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.632480 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(4aa944cd0b82429da02bd3dd35e0cce7) complete. Timing: real 0.028s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":1279,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":73,"threads_started":1,"wal-append.queue_time_us":9408}
I20240502 14:03:23.634073 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7): 4925 bytes on disk
I20240502 14:03:23.635212 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.637413 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 14:03:23.638229 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d): perf score=0.033379
I20240502 14:03:23.645138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.671597 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.676600 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.698770 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.700021 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.710013 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.071s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1274,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1223,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":36407}
I20240502 14:03:23.710938 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d): 4840 bytes on disk
I20240502 14:03:23.711644 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(a476d1664de7456ab21ddbf2e12b4f0d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 14:03:23.714593 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.716665 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.747247 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.761411 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(20e960d4e55047918a1dad82fae21ecd): perf score=0.033384
W20240502 14:03:23.773204 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.773910 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:23.776103 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.783260 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.783565 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.801338 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.809183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.818902 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.820595 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.059s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":20650,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":5334,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":20269}
I20240502 14:03:23.821437 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd): 4840 bytes on disk
I20240502 14:03:23.822865 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(20e960d4e55047918a1dad82fae21ecd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 14:03:23.873780 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.878285 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:23.887948 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.893486 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.897495 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:23.900004 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:23.908617 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.919822 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:23.920517 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:23.945478 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:23.950240 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:23.955859 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:23.957027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:23.958573 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:03:23.959213 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:23.970849 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:23.992064 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:23.992451 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.017354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.022121 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.023057 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.025324 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:24.062345 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.076026 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(d9e0261898a445949f3cf21d2efb599f): perf score=0.033338
I20240502 14:03:24.088665 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.094187 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.099088 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.104606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.114393 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.123276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.132376 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.056s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":27923,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1416,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":550,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":237}
I20240502 14:03:24.133428 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f): 4840 bytes on disk
I20240502 14:03:24.139011 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(d9e0261898a445949f3cf21d2efb599f) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":5220,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
W20240502 14:03:24.148015 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.164499 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.169268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.169883 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.173012 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.173345 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.180246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:24.183145 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.188550 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.191154 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.214260 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.214515 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.214991 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.240813 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:24.286727 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 14:03:24.307791 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.338308 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.342955 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.349143 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.354947 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.358531 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.367210 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.370651 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.384529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.384528 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.414937 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:24.417119 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.423754 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.428632 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:24.437255 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:24.454223 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:24.454553 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.468917 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:24.477108 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.499897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.504838 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.507972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.509410 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.521979 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.526463 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.531174 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:24.534838 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:24.545207 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.545666 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:24.546314 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.557255 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.584154 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:24.595175 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:24.636518 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.650172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.650274 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.655644 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.660868 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.663115 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.676247 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.684895 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.694957 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.706200 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.707566 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.713317 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.722023 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.722553 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.725204 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.731009 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.746023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.753175 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.815824 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:24.817183 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.817471 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:24.819059 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:24.846748 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:24.874660 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.883860 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.886651 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.901629 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.902509 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:24.912598 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:24.915444 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:24.922528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:24.925144 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:24.981120 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:24.998955 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:24.999306 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:24.999788 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.007128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:25.009991 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:25.019126 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.026604 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.028765 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.038388 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.062812 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.063213 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.078562 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.084060 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.149883 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.156925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.161398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.162534 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.164059 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.185343 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.192023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.208740 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.212097 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.219542 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.220367 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.233352 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:25.282467 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:25.308032 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.317315 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.325275 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.328743 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.338321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.351505 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.367794 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.385798 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.388893 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.391355 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:25.395411 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:25.405098 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.405503 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.453727 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.457387 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.464902 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:25.465059 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:03:25.465167 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:25.475751 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.483119 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.484697 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:25.487715 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:25.491650 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.492635 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.496520 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.502522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.505352 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.512800 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:25.519465 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:25.592777 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.594410 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.594913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.596653 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.605883 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.609057 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:25.618438 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:25.657321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.658674 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.662899 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.662971 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.667145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.676429 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:25.684101 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:25.726305 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:25.758138 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.770035 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.771749 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.774806 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.775552 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.781155 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.800119 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:25.821260 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:25.827210 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.832163 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.835846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:25.849179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:25.878028 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:25.930965 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:03:25.950104 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:25.954424 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:25.954489 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:25.970742 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:25.972770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:25.973443 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 14:03:25.984954 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:25.988727 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.036460 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.041688 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.042613 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.044549 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.044831 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.045724 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.087965 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.091079 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.092231 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:26.097384 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:26.116446 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.119303 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.126974 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.128206 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.138053 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.144120 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.146353 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.154925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.158151 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:26.174603 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.207675 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.219774 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.223345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.234285 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.234376 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.268723 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:26.302950 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:26.314412 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.321494 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.322782 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.334795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.371254 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.380113 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.389540 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.389940 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.399315 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.405849 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.411963 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.419693 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.425539 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.432715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.435122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.436558 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:26.442526 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.451622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.451690 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:26.496347 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:26.510394 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.511365 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.519953 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.529104 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.529613 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.532054 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:26.579972 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.611371 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.615254 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.615566 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.624467 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.625691 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.645078 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:26.657727 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:03:26.658357 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.673509 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.682238 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.683197 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.697352 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:26.701014 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.701532 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.702795 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.712658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.714270 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.714969 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.728338 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.736084 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.744163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:26.771251 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:03:26.783505 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 14:03:26.816882 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.848562 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.862588 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:26.866269 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:26.866783 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.869493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:26.882522 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.885058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.894486 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:26.902510 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:26.915421 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.919606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.929863 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:26.946995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.949146 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.954020 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.956907 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:26.963675 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:26.970657 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:26.987396 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:26.989323 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:26.992650 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:26.999212 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:27.020561 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:03:27.030030 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:27.035800 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.041929 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.050984 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.052098 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:27.065490 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:03:27.067039 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:27.072644 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.073531 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:27.100698 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:27.124933 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:27.150810 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.151530 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.164469 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.167059 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.168076 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.175293 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.186250 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.192418 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.196571 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.213732 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.221715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.222844 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:27.252079 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:27.271849 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.280527 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.286356 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.295869 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.299715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.310078 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:27.319330 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:03:27.374852 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.375613 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:27.377797 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:27.380100 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.380810 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.384678 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:27.385561 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:27.407655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.439265 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.440946 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.442472 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.448190 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.448608 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.461683 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.462612 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.476413 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.492710 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.494102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.526808 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.527388 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.593952 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:27.595577 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:27.597580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.613314 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.624264 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:27.625108 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:27.625569 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:27.630879 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:27.643685 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:27.656554 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.686709 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.693969 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:27.702039 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 43 similar messages]
I20240502 14:03:27.713387 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.714231 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.715165 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.737934 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.743302 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.759588 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.763221 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.771095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.778398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.795089 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.802961 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.808753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.814455 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.814716 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.823302 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:27.871098 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:27.873821 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.881106 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:27.897810 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.907480 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:27.913625 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:27.913921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.918239 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:27.924367 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:03:27.937794 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:27.990869 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:27.992954 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:27.996904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:27.999765 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.019569 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.022327 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:28.043082 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:28.064590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:28.065611 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:03:28.067974 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.068732 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.072111 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.089030 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:28.102918 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:28.124485 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.137099 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.139241 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.146027 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.156831 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.157900 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.162685 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.172412 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.185496 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.186512 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.191382 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.193467 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:28.196154 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:28.207238 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:28.312455 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:28.335669 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.336462 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.341548 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.344012 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.350253 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.352917 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.359254 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.366890 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.374917 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.377611 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:28.378455 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:28.388026 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.426268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.428138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.431830 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.441020 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.451107 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.451341 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.464342 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.469769 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.470858 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:28.471657 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:28.474973 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.477556 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.490887 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.501819 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:28.535692 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:28.571130 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:28.624488 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 14:03:28.625115 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.626348 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:28.632009 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:28.635174 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.636849 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.639463 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.648108 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.663460 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.665819 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.665962 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.674995 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.675154 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.679673 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.686952 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.689919 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.691676 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.719672 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.726213 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:28.733984 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:28.744825 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.838580 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.840721 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.844046 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.847128 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.851892 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.876767 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.906399 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.908427 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:28.916038 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:28.918128 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:28.935564 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:28.935874 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.943048 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.948530 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:28.953425 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:28.957764 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:28.958654 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:28.974126 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:28.976732 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:28.979406 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:28.991868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:29.033229 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:29.045995 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.049317 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.052223 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.057472 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.068137 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.078235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.147840 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.148337 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.154192 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.156261 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.164882 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.170838 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:29.172781 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:29.194733 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.196235 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.206012 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.208348 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.210532 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:29.217532 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:29.236466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.236945 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.250713 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.251251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.253803 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.268414 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.274758 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.311487 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:29.358397 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:29.380049 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:29.391510 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.422912 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:29.424391 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.425622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.427392 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.430088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.466434 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.478909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.479569 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.484474 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:29.487689 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.493219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.499332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.501544 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.503237 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.504608 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.508867 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.517060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:29.519994 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:29.524484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.527138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:29.533756 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:29.544402 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.545607 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:29.550464 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.555032 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.558482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.568629 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.570578 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:29.588375 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:29.592777 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.622196 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.629350 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.639093 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.639505 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.647379 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.664083 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:29.687561 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:29.724668 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:29.729154 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.730669 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.735288 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.736593 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.743822 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.748342 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.811903 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.814821 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:29.823027 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.825665 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.827786 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.834110 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.843036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:29.847967 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:29.871726 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:29.876372 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:29.897536 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:29.898470 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:29.917058 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:29.991742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.006150 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.020331 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.021320 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:30.023582 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:30.034343 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.034415 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.044230 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.045234 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.045377 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.052402 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.054339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.056671 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:30.073596 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:30.112493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.113422 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.113615 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.130755 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.131599 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:30.140754 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:30.149755 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.161718 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling FlushMRSOp(4b1230a7230f48558da1069a5988b981): perf score=0.033350
I20240502 14:03:30.189282 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: FlushMRSOp(4b1230a7230f48558da1069a5988b981) complete. Timing: real 0.027s user 0.011s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":9545,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1468,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":482,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6589,"threads_started":2,"wal-append.queue_time_us":687}
I20240502 14:03:30.190609 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981): 4586 bytes on disk
I20240502 14:03:30.191023 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240502 14:03:30.197607 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:30.219144 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.222741 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.232805 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.236775 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.238335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.246577 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.266613 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.283322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.298975 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.303831 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:30.304432 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:30.307821 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.320631 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.333437 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.335711 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.337082 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.347364 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.356621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.360083 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:30.405246 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:30.437196 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:30.438521 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:30.442014 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.444397 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.453485 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.453904 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.460085 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.495859 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.500942 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.507588 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling FlushMRSOp(81b2d5d0f4044106bed079c3faa72447): perf score=0.033396
I20240502 14:03:30.510584 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.513126 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.541260 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.543283 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.543808 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:30.549674 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:30.554555 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.566778 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.568317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.575698 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.579391 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.586652 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.588313 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: FlushMRSOp(81b2d5d0f4044106bed079c3faa72447) complete. Timing: real 0.081s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38983,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":2171,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":17678,"threads_started":2,"wal-append.queue_time_us":19957}
I20240502 14:03:30.589184 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447): 4925 bytes on disk
I20240502 14:03:30.595506 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.599004 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.599619 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 14:03:30.603219 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.605829 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:30.622018 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:30.643030 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:30.680490 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:30.728479 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 14:03:30.757560 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.761373 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:30.769104 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:30.775645 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.776316 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.789059 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.790112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.799332 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.801278 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.801999 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.808379 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.816747 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.822016 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.825940 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.834030 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.845855 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.863390 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:30.867167 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.893712 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:30.925491 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:03:30.926409 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:30.935431 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:30.946009 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:30.954275 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:30.962684 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:30.970968 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:30.971287 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:31.033445 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:31.084079 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.084832 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.098167 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.099663 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.109308 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.122444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.127768 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:31.130439 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:31.139257 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.139391 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.147456 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.156914 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.177827 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.191318 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.191699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.199535 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.201602 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.209285 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.219907 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.274984 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.277115 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.290326 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.291663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.294596 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.295998 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:31.326423 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.329504 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.332166 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.342254 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.343545 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.350764 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.365427 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:31.367177 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 14:03:31.416507 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.424329 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.427911 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.435604 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.436172 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.437824 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.444154 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.453401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.454589 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.458747 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.483197 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.492070 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:31.493598 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:31.496739 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:31.513291 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:31.517735 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.562933 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.566560 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.571462 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.594727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:31.600675 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.602825 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.616994 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.627627 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.628504 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.633026 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:31.636231 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.645347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.646253 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.664335 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:31.681931 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:03:31.706414 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:03:31.723520 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:31.724862 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.728804 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.738593 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.739403 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.747431 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.755004 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.823756 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.838193 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.842165 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.845909 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.856446 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:31.866604 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:31.871173 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:31.879408 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.885187 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:31.892740 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:31.901625 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:31.909593 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:31.925774 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:31.983386 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:31.983657 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:31.987457 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.001727 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.008131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:32.022560 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:03:32.023075 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.026399 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.038874 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.040309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.047307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.048525 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.048619 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.057113 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.061389 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:32.064100 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:03:32.095105 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 14:03:32.100941 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.103356 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.107474 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.126766 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.128207 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.136395 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.152235 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:32.178022 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:03:32.195451 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.196395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.199163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.199656 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:32.224738 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.225499 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:32.227591 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:03:32.232302 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:32.251220 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:32.255988 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.259999 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.261687 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.263600 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.272614 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.304075 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:32.310813 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.333467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:32.333551 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:32.339622 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.341058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.341631 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.346658 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.362226 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:32.392894 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:32.399461 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:03:32.418268 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:32.422822 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.424379 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.432058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.433651 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.437824 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.453328 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:32.455233 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.501521 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.503194 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.508433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.510820 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.517711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.520316 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.560226 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.560391 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.569605 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.575152 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.582278 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.598788 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.617813 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.620422 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.622231 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.624433 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.648994 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.650102 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:32.719823 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:32.736665 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.749167 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.752516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.755124 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.760982 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.771836 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:32.806217 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:32.809563 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.822860 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.832484 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.843583 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.849063 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:32.851061 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.865520 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:32.889214 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:03:32.891304 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:32.894378 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.899935 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.900439 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.904016 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.911844 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:32.912680 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:32.916817 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:32.925676 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:32.926637 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:32.932515 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:32.945976 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.980682 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:32.988133 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:32.991381 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:03:33.025759 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:33.025857 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.030804 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.032531 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.035321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.056135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:33.082021 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:33.109244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.114197 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.114883 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.126353 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.126616 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.131821 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:33.148568 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:03:33.153446 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:33.162066 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.162818 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.164500 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.165237 10464 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:33.165339 10464 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:33.165796 10464 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:03:33.166452 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:03:33.166580 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:03:33.167237 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:33.167352 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:33.167443 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:33.167577 10464 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:33.173394 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.176551 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.182111 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.249469 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.261544 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.271986 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:33.272725 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:33.281677 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.281733 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.314992 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:33.328583 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:33.339283 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.340466 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.346208 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.348188 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.358584 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.365252 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:33.398598 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:33.424535 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.429157 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.430547 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.432412 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.437587 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.448817 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.453119 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.473994 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.478976 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.480940 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.481240 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.482998 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:33.509555 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:33.527748 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:33.546852 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.556874 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.558305 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.559494 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.565373 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.566522 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:33.681404 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:33.692513 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.696152 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.700204 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.713277 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.715339 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.715597 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.728730 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:33.730240 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:33.730796 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.733903 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.734514 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.742671 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.753507 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:33.755889 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.757937 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.776723 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.782750 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.810300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.816568 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:33.865942 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:33.876258 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.884361 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.889189 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:33.904086 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:33.911669 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:33.914007 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.915558 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:33.936439 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 14:03:33.943063 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:33.960124 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:33.960656 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:33.962193 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:33.963068 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:33.970588 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:33.971364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.020339 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.025246 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.032814 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.035369 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:34.038022 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.039995 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.046540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.121635 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.123082 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.127708 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.133865 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.138430 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.152160 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.185067 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:34.187616 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.190016 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:34.191358 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.193850 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.195014 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.196650 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:34.205653 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:34.225023 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.236590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.238813 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.252516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.259503 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.261693 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.262445 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:34.269032 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:03:34.303542 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.382678 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.388855 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.401860 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.416569 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:34.423969 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.438686 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.441891 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:34.442663 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:34.453224 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.465778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.465803 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.467145 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.467190 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.478796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:34.481784 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.483593 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.484941 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.492892 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:34.495110 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:34.495677 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.498222 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.510216 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.526101 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.534030 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:34.549659 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.552138 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.557543 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.570019 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.576586 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.578846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.583606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:34.599404 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.600785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.604254 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.616259 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.622078 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:34.623939 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.712647 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.718364 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.726050 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.735044 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.742498 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:34.743643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:34.761978 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:34.780090 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.790951 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:34.792840 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.806414 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.819308 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.823297 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:34.825537 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.846765 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:34.851480 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:34.854064 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:34.865588 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:34.883589 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:34.885115 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:34.885599 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:34.920897 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:34.950229 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:34.978011 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.005487 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.021206 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.028026 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.037988 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.038450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:35.049391 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:35.051875 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:35.056123 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:35.063418 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.074612 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.091080 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.098609 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.101693 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.104468 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.104542 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.105159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.111941 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.115484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.116185 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.123402 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.135514 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.148818 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.158519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.159472 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:35.161439 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:35.172432 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:35.221246 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:35.300552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:35.301077 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:35.302696 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.310566 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.313139 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.346236 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.358304 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.358922 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.363610 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.369108 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.382550 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.382776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.395958 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.397876 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.398936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.404117 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.420603 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.421793 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.425778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.480567 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.483719 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.489379 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.492784 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.501367 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.512938 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:35.556128 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:35.561450 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.569079 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.570583 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.573851 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.574476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.586122 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.608556 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:35.612430 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:35.621750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.624006 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.636384 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.636778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:35.645213 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:35.649078 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.759486 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.769548 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.772439 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.785825 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.788797 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.795678 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.796828 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.805586 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.813367 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:35.818823 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:35.823696 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:35.835786 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:35.838038 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:35.840417 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 14:03:35.850828 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:35.857290 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:35.875568 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.878964 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.896401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.897895 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.906030 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:35.974562 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:35.990760 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:35.991257 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:35.991479 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.002395 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.031564 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.071223 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:36.080243 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:36.081444 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.081454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.083263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.088392 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.120012 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.140525 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.144568 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.146308 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.158914 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.161448 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:36.167574 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.176698 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.184737 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.191145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.191759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.209913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:36.215548 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:36.219282 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.226778 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.241971 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.248209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.253734 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:36.260537 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:36.263417 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.278082 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.278731 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:36.325309 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:03:36.352520 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.357599 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.359678 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.366796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.378856 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.380982 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.398535 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.429273 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.430395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.437883 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.452757 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.454591 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.460850 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:36.470865 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.481109 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.485127 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.485570 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.487996 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.491408 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.504446 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:36.549700 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 14:03:36.583894 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:03:36.593605 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:03:36.596697 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.621281 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.622870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.626181 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.627363 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.629420 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.631264 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.631855 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.637125 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.637488 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.645576 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.660658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.662654 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.729929 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:36.730749 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:36.736385 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.744058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.745378 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.750591 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:36.754928 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.760236 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.775491 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.783589 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.789579 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.792884 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.794124 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.801920 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:36.808794 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:36.853343 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:36.877442 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:36.879540 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:03:36.885633 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.890312 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.891861 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:36.905702 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:36.912642 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:36.927098 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.930688 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:36.975440 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:36.987483 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:36.988327 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:36.993393 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:36.999222 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.029449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.043975 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.048214 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.048236 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.048241 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.051702 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:37.062108 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:37.064683 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:37.102149 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 14:03:37.104125 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:37.136363 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.147254 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.151021 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.151168 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.153846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:37.158138 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:37.161193 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.164260 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.172506 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.182444 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.183840 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.202174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:37.210865 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:37.228508 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:37.238236 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240502 14:03:37.286366 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.291095 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.294947 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.301179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.312347 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.323925 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:37.332005 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:37.366312 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:37.371273 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.372088 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.373224 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.382234 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.386969 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.391196 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.391288 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.391949 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.396624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:37.397378 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:03:37.399655 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:37.406828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.409256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.419982 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:37.489594 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:37.512348 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:37.537477 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.542069 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.551268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.551522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:37.561748 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:37.563149 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:37.571579 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:03:37.574628 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.577293 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.577589 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.579538 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.587970 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.588188 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:37.606046 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:37.608815 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.627763 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.632930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.638832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.640784 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.640869 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.665635 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.720541 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.745082 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.747015 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:37.754830 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:37.762979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.763301 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.778136 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.899654 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.911588 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.912588 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.912860 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.918270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.925750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.930505 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:37.930876 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.932708 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:37.937332 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:37.939007 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.947162 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.954785 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.955319 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.959787 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:37.966786 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:37.969228 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:37.971463 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:37.979259 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:03:37.994578 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.018294 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.028620 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.033025 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.045910 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:38.063161 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:38.064816 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:38.073346 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:38.106123 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:38.108999 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:38.110613 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:03:38.114893 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.117991 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.118520 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.118953 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.121325 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:38.185096 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:03:38.185988 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:38.229745 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.229750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.244603 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.246461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.247283 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.251283 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.279457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.287345 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:38.290820 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:38.292301 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.292707 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.300343 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.311823 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.339673 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.341413 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.344602 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.355271 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.356607 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.359779 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.446138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.449558 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.459373 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.459686 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.461344 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.483520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:38.496412 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:38.499559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.499559 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.503717 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.504325 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.508002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.532032 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:38.621444 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:38.639516 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:38.649044 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:38.669804 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.672703 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.686070 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.700393 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.710952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.716713 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.717664 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.718045 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:38.718380 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:38.719785 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:38.721097 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.725466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.733280 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.740547 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.740777 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.779381 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.781576 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.789639 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.794142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.801891 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.803512 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:38.823187 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:38.827649 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.835458 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.835875 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.843281 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:38.868240 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:38.888895 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:38.890811 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:38.901108 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:38.902237 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:38.902314 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:38.923499 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:39.000186 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:39.002986 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.008757 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.016331 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.024955 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.026894 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.026943 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.030742 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.031358 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:39.050071 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.061051 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.064802 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.064801 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.078312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.117427 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 14:03:39.130518 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.133854 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.135394 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.145408 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.146157 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.153985 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.167510 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:39.241228 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:39.247459 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.256371 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.260426 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.267757 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.272899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:39.276592 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.294059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.310449 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.318840 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.343398 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.344872 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.350230 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.356170 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.363431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:39.366957 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:39.367906 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.369163 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.376758 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:39.377669 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.381434 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.384306 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.387061 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.404436 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.408155 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.413879 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.421588 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.427659 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.436147 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.523865 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:39.524945 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.531239 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:39.532725 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:39.532739 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.534196 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.545120 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.546641 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:39.547144 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:39.552253 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:39.582764 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.642578 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.643453 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.653388 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.656379 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.659603 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.661103 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.672010 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.678828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.683952 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.688647 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.692831 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.700982 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.712481 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.727892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.732429 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.741006 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.749473 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.753803 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:39.754094 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.758630 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.769755 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:39.773811 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.778679 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.795951 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.797125 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.805282 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.809419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:39.854285 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:39.937153 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.940536 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.948799 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:39.952845 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:39.957073 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:39.960899 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.965202 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.967150 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:39.989336 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:39.991045 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:39.991518 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:39.991796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.016515 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.017918 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.019640 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:40.023224 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:40.028970 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.047242 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.048446 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.053205 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:40.061880 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:40.083039 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:40.114928 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:40.165807 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:40.181843 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.186561 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:40.188509 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:40.197867 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.199651 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.199688 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.222610 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.224895 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.229140 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.240665 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.251950 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.255673 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.262251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.272497 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.273662 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.282135 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.284570 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.299556 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.299612 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.439852 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.439937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.446641 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.455664 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:40.460018 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:40.465322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.483218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.483678 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.492827 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.493278 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.494298 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.497328 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.498173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.508198 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.508520 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.513813 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:40.520923 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:40.529027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.531078 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.532414 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:40.548380 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:40.550172 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.551602 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.564731 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:40.572315 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:40.585139 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.586166 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.598264 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:40.657239 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:40.725817 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.728852 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.729487 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.732389 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.752547 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.766885 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.774410 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.780293 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.781082 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.789371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.807572 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.809656 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.859294 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.859294 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.861814 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.870210 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.871874 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:40.872211 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:40.874486 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:40.877799 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:40.892793 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:40.893967 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.912753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:40.917435 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:40.935402 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 14:03:41.023262 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:41.023517 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:41.028465 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.031548 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.036938 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.039258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.042943 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:41.069912 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.114178 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.125627 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.127597 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.134919 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.136713 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.138974 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.147883 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:41.154487 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:41.154474 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.163880 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.164474 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.176113 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.182122 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.191202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.191582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.204592 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.205351 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.205750 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.219035 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:41.267261 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:41.277748 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.319607 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.321640 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.326406 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.329794 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.333266 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.347335 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.398664 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.402916 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.404254 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.406397 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.406427 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.410023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.415006 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.417737 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.423298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.436053 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.441699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.448758 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:41.468515 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:41.474287 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 14:03:41.486878 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:41.533387 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:41.565119 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.575162 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.582655 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.599320 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.611563 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.621683 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.628477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.641036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.654039 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.662901 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.662992 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.674921 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.680879 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.682793 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.686892 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.687055 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.693370 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.693584 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.706305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.707180 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.725914 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:41.728797 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.749606 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.749934 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.752817 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:41.758428 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.879911 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.880723 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.889317 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.896082 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:41.899372 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.901280 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.910959 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:41.911537 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:41.912483 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:41.917779 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:41.921046 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.922339 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.922629 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:41.941411 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:41.942168 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:41.956038 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:41.956447 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:41.964577 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:41.971611 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:41.981863 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:41.983870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:41.994139 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:42.002605 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:42.009980 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.011147 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:42.091817 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.112943 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.128208 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.128487 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.129709 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.132516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.141886 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:42.152335 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.156963 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.158923 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:42.170753 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.193049 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.198643 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.199671 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.202818 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.205384 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:42.211959 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:42.223729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.236089 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.239190 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.248214 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.253479 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:42.260506 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240502 14:03:42.284390 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.286734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.294484 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.298836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:42.301968 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.304646 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.314826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:42.347378 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:42.399088 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:42.409452 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:42.421787 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.431452 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:42.434489 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.437849 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.451010 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.458885 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.473708 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.473886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.483466 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.490561 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.499364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.499356 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.506424 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.506672 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.512068 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.518241 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.520902 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.524247 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:42.527141 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:03:42.527895 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:42.536638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.549628 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.556975 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.561280 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.562978 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.564333 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.592913 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:42.597266 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:03:42.612939 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 14:03:42.700040 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:42.706619 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:42.716562 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.720615 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.727789 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.729260 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.729871 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.752779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.753207 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:42.754554 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.766784 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.772508 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.783880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.799007 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:42.803419 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:42.803771 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.824218 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.833951 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.838361 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.842383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:42.854517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:42.889499 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:42.937986 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:42.948719 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:42.949323 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:42.949671 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:42.950399 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:42.964416 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:42.967195 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.017168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.021832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.022863 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.022974 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.039952 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.061260 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.094484 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:43.095845 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:03:43.100908 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.107589 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.110018 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:43.125888 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:03:43.127866 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.129365 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.135309 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.136263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.138439 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.144770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.163938 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.177456 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:43.196290 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:43.225953 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.231045 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.234124 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.235720 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.239785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:43.246251 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:43.260447 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:43.291286 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:03:43.336374 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.354861 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.357851 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.359086 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.360071 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.369194 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.376226 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.377317 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.380833 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.381093 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.384481 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.390914 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.402926 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.409862 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.414352 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.435721 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.438226 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:43.446272 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:43.447453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:43.453507 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:43.482939 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:43.484885 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:43.488449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.493865 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.498271 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:43.499457 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:43.504228 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.507688 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:43.557031 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:43.593561 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.596422 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.600946 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.606514 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.623075 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.635658 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.645843 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.656483 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.660068 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.677204 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.680402 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.685962 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.728601 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.740383 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.744438 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.746345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.759488 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.778648 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:43.820622 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:43.850399 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.860431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.882790 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.883715 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.885476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:43.888721 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.908159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.910979 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.911628 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.920744 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.921268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.926961 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:43.938431 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:43.940384 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:43.947088 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:43.949959 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:43.952468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:43.962642 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:43.980469 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:44.005496 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:44.048719 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:44.063081 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.075188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:44.075798 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.079077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.080780 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.090286 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.096897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.137421 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:44.145309 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 14:03:44.154487 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.185758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.189332 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.192230 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.203531 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.214859 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.230494 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.235292 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.238759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.239583 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.240166 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.240422 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.253899 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.267344 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.268494 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.268687 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.293853 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.306090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.316304 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.316473 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.321234 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.322399 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:44.333814 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:44.342823 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.347314 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.358603 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.361992 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.446926 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.450374 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.462770 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:44.463183 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:44.472657 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.478976 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.485827 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.513773 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.515607 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.518604 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.519266 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.524181 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.547926 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.550803 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.551544 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.555845 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.556577 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.557804 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.563836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.592871 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:44.615516 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:44.656250 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:44.695987 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.696524 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.706012 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.708297 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.713425 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:44.740784 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:44.746274 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.749459 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:44.750667 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.770004 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.781301 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.787856 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.791925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.796828 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.804714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.808071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.823967 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.840782 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.842443 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.842677 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.870445 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.874215 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:44.876662 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:44.878829 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:44.883008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:44.889870 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:44.903116 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.915657 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:44.938244 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.941087 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:44.946480 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:44.979790 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:44.981303 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:44.982982 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:44.991945 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:45.001366 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.020567 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.032413 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:45.036316 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 14:03:45.068810 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:45.076187 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:45.095995 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.117461 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.118218 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.131863 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.141744 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.148172 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.156801 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.162822 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.164875 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.164964 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.165585 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.168138 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.170961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.194196 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.200722 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.204486 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.209303 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.213325 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.224536 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.234771 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.236922 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.240996 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:45.248804 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:45.258575 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.269364 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:45.321864 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:45.354661 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.356318 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.358287 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.359580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.360040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.378954 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:45.386962 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:45.458405 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.460529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.463567 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.466322 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:45.468446 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:45.471140 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.474750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:45.542450 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:45.548245 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.553063 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.554252 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.562387 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.567171 16777 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:45.567283 16777 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:45.567821 16777 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:03:45.568764 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:03:45.568899 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:03:45.572587 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:45.572698 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:45.572736 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:45.572939 16777 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:45.584880 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.590629 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:45.628646 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:45.630115 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:45.688637 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.690878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:45.693619 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:45.695190 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.695819 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.708880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.710819 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.711051 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.720191 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.725749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.732003 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.732303 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:45.744984 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:45.753897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.779062 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.779100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.782503 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.790791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.796044 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.800367 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:45.850651 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:45.904893 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.907135 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:45.911288 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.915678 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:45.922652 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:45.958081 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:45.981707 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:45.993527 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:45.997433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.001564 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:46.017522 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.033097 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.037933 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.047521 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.057716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.069059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.086338 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.100451 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.101212 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.108759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.113790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.119048 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.124842 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.130178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.140486 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.141500 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:46.156281 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:46.159919 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.171139 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.171666 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.172649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.173509 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.248576 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.249222 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.257669 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.259881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:46.263229 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.273414 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.278820 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:46.303869 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:46.364692 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:46.367092 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.370486 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.376761 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.377022 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.385135 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.386906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:46.406844 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.438815 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.447419 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.482614 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.483706 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.486136 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.489596 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.493094 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.504793 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.508965 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.512605 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.514182 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.516042 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.525205 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.543423 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.543641 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.559029 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.562033 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.586908 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.622601 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:46.624833 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.641582 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.643132 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.646953 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.651908 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.664770 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:46.685492 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.739441 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.742883 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.743443 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.748915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:46.757635 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:46.762741 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.763355 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.798990 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.801752 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:46.803488 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 14:03:46.803884 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:46.822391 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.828621 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:46.847340 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.856979 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:46.858294 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:46.870141 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:46.870235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:46.877799 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:46.883692 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:46.887676 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:46.915589 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:47.001664 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.007818 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.017083 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:47.020589 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:47.025204 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.038981 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.050904 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.051738 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.055970 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.065757 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.067713 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.068241 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.069701 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:47.070626 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:47.082850 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.084792 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.094424 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.097219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.097816 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.104261 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:47.111624 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.121547 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.240067 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.273258 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:03:47.274441 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:47.280542 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:47.281158 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:47.291574 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.291872 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.297086 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.310235 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.317519 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.318352 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.325201 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.326656 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.327828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.327867 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.337715 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.339052 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.345069 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.347947 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.348304 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.351158 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:47.360899 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:47.382001 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.383690 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.392468 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.405829 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.407927 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.413663 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.428380 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:47.464547 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.501446 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.509352 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:47.570293 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:47.570369 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:47.571360 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.589684 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.598543 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.600298 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.602363 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.612974 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.620605 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.625919 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.631486 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:47.634465 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:47.638677 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.657374 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.658556 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.667982 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.670120 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:47.681305 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:03:47.683601 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:47.693468 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:47.696825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.706879 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.708482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:47.729576 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.737268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.745738 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.746793 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:47.751438 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:47.754388 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:47.762900 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:47.796555 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:03:47.887423 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:47.901429 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:47.910352 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:47.910367 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:47.914685 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.924635 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:47.940553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.940847 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.947958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:47.949957 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:47.958446 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:47.959995 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:47.960994 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:47.990578 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.003000 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.005286 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.007705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.010586 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.018965 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.021319 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.026021 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.028306 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.034109 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.038950 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:48.049098 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:48.057353 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.083535 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.084609 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:48.107570 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:48.181309 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 14:03:48.185137 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:03:48.188930 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.191727 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.198794 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.211117 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.211695 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.214219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.218276 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:48.226495 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:03:48.233116 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.235453 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.259800 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.260661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.277534 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:48.291626 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:48.337597 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:03:48.345978 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.350273 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:48.361876 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:48.372078 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.375227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.389358 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.411087 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:48.417831 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:48.420107 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.432126 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.433493 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.434536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.434747 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.439013 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.519413 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.528555 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.540235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.542527 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.559705 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.571190 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.578393 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.578555 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.585189 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.588888 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.600320 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.600327 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.603552 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.618784 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.620786 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.625912 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.629968 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.642745 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:48.669512 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:48.755895 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:48.764053 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:48.765025 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.775461 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.791393 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.799979 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.815860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.818316 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.818756 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.826442 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.840428 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.849021 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:48.852429 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.854430 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.856168 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.863727 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.866155 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:48.873201 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.880172 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.885157 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:48.957653 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:48.959538 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:48.962580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:48.965662 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:48.971128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:48.972390 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.009627 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:49.078379 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.116570 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.124588 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.131461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.131739 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:49.132732 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:49.136950 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.138082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.144173 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.146170 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:49.162338 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.166265 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.167349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.169597 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.172573 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.176760 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.187418 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.187942 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.189977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.215245 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.216023 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.222339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:49.239537 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:49.244138 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:49.270247 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.277683 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.277691 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.280125 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:49.289619 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:49.297120 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.298416 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:49.317621 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 14:03:49.383852 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.384828 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.387944 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.417069 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.419183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.434691 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.439996 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.440235 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.444872 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.452740 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.456959 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.468573 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:49.505888 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.508886 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.511727 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.523695 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:49.526347 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:49.527267 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.532399 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.534281 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.538445 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.554672 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.564067 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.564908 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.579663 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.591574 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:49.597009 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:49.634625 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.658830 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.659744 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.666863 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.673290 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.673924 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.686856 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:49.693531 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:49.712440 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:49.718374 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:49.746126 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.749955 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.752475 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.755668 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.758550 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:49.776751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.788597 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.802618 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.819348 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.820858 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:49.825177 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:49.834858 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:49.841858 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 14:03:49.877382 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:49.957930 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.960072 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.978855 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:49.979130 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:49.984426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:49.987787 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.009716 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.020351 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.020820 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.023663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.024533 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.033017 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.035955 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.039167 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.040705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.057442 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.058605 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.066906 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.076915 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.095330 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.096961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.116072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.122673 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.133327 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:50.147401 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:50.219048 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.219425 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.220167 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:50.222815 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:50.228962 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.235002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.241153 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:50.253862 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:50.265991 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:50.284227 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:50.292086 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:50.303715 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:50.304924 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.318359 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.319299 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.319914 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.328475 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.329784 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:50.336619 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:50.340667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.340713 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.348031 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:50.361258 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:50.370330 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:50.381304 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:50.385155 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:50.435500 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:50.456343 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.463441 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.465879 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:50.474597 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:50.475893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.489593 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.500332 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.502302 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.508811 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.509465 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.517398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.519320 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.532238 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.590070 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.615514 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.620803 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.626478 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.631112 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.663632 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:50.706780 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:50.729048 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.729847 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.730697 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.731317 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.732549 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.742625 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.742870 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.750066 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.754252 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.765455 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.780907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:50.791677 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:50.792727 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.793877 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.807823 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:50.812089 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:50.823559 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:50.838649 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:50.839493 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.846591 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:50.847980 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:50.855640 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:50.857580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.859504 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.876425 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:50.883631 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:50.910874 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:50.926194 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:51.013810 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.021513 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.023188 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.032202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.034127 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.040181 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.043031 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.056545 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.057611 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.058563 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.062002 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.078734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.079860 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.090077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.095729 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.100718 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.108780 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.129914 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.138538 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.147096 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.169214 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.172451 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:51.175462 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:51.180900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.192710 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.273001 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.281445 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:51.281711 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:51.284246 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.300097 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.304657 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:51.313160 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:51.318580 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.318892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.328333 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.333202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.352121 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.354194 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.356815 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.420044 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.427012 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.431660 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.432731 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:51.440769 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:51.445346 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.450498 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:51.475427 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:51.491359 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:51.505699 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.510607 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.511557 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.514168 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.526991 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.536185 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.550257 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.550904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.552654 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.559409 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.566138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.572261 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.675623 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.681345 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:51.682142 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:51.684299 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.698312 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.711936 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.712194 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.714128 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.715940 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.728523 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.731644 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.732704 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.744083 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:51.765180 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:51.787978 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.793969 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.799635 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.801494 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:51.805519 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:51.823093 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:51.836076 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.848681 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.849555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:51.851341 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:51.863852 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:51.882581 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:51.882696 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.883138 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:51.907433 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:51.949249 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 14:03:51.986392 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:51.988961 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:51.991711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.004941 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.014138 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.023094 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.031817 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.036835 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.041739 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.052345 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.054504 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.068912 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:52.069866 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:52.089732 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:52.105181 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.106536 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.116724 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.121029 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.123368 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.123883 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:52.128624 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:52.167202 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:52.179062 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:52.204077 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.210855 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.212347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.219839 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.222067 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.227913 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:52.241756 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:52.297488 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.310803 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.316974 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.319839 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:52.324450 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:52.327128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.333627 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.333659 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.339730 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.342468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.348451 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:52.349375 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:03:52.351608 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.352931 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.365383 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.367187 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.390758 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.402673 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.409722 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.413591 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:52.457345 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:52.513823 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.516945 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.517300 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.529559 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.532415 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.542207 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:52.561018 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:52.572245 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.575052 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.577984 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.607249 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.612141 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.615418 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.619622 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.626488 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.629704 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.636638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.640030 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.651115 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.658638 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.663676 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.665340 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.674738 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.696074 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:52.703383 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:52.714373 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:52.738534 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:03:52.739956 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:52.755911 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:52.828387 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.830745 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.836536 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.837196 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.843868 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.844401 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:52.853212 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 14:03:52.855659 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:52.866693 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.881495 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.885221 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.901055 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:52.908715 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.908973 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.921487 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:52.922389 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:52.925318 21013 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:03:52.925428 21013 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:03:52.925909 21013 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:03:52.927920 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:52.928025 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:52.928068 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:03:52.928197 21013 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:03:52.928529 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:03:52.928676 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:03:52.937662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:52.939592 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:52.943599 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:52.954721 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:52.962235 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:53.025182 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:53.031420 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.081154 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:53.089583 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:53.099877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.104652 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.108441 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:53.108628 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:03:53.109930 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.115593 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.118094 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.120237 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:53.130712 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:53.132721 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.133423 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.137830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.146095 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:53.146330 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:53.166754 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.169842 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.171105 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.176461 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.197048 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.197347 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.207332 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.222532 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.233181 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.246927 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.260222 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.266708 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.288985 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:53.331360 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 14:03:53.332104 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:53.358436 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.362809 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.379021 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:53.386941 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:03:53.387068 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.392442 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.397341 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.448752 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.450582 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.459086 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.475916 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.478967 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:53.483883 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:53.498768 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.505030 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.505559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.505640 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.509235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.523154 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:53.525489 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.528559 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.549412 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:53.554602 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:03:53.558348 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.569061 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.572176 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:53.578966 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:53.579185 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.580888 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:53.640589 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:53.658149 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.665365 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.666019 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.669562 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.670445 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.687414 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.732777 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.741017 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.744526 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.755820 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.758430 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.763749 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.764323 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:53.785112 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.786188 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.786406 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.803793 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:53.806326 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.812302 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:53.830015 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:53.921574 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:53.929313 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:53.949057 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:53.952500 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:53.964138 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.977695 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:53.990082 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:53.995463 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:53.996058 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.004446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.007555 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.014098 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.015580 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.019526 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.038970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.055060 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.056670 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:54.059265 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:54.063174 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.064101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.082000 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.126165 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.126205 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.137027 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.139760 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.148984 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.179332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:54.188561 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.201015 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.203675 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.205217 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.209100 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.211810 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.218945 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:54.284669 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:54.290292 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:54.295876 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.306026 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.307065 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.315608 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.321041 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.322793 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.326416 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.333401 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.345292 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.348681 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.359783 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.365046 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.368942 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.416467 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.424731 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.429594 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.430756 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.443603 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.461371 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:54.464207 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:54.484386 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:54.512638 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.516993 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.518389 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.519845 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.521320 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.522010 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:54.534863 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:54.577335 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.585237 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:54.588164 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.588724 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.591660 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.592875 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.600872 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:54.634421 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.638283 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:54.638617 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 14:03:54.645591 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.651023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.651227 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.652369 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.653357 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.753048 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.757300 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.759596 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.760826 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:54.761363 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:54.762661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:54.777746 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.796849 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:54.801944 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:54.837044 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.841615 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:54.845989 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.854843 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.860162 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.878425 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.906504 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:54.917234 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:54.917325 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:54.920604 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:54.922806 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:54.924072 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:54.943888 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:54.999027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.000622 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.012878 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.022177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.022490 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.028664 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.044646 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.057606 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.057806 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.061127 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.068759 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.085260 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:55.105237 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:55.110299 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:55.157307 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.160899 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.163491 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.163688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.166214 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.174417 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.209743 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.211757 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.225497 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.227133 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.238673 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.239004 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.254803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.269259 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.278103 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.289973 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.291188 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.303877 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:55.343848 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:55.353839 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.360760 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.366904 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:55.368757 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:55.379619 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.404594 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.409175 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:55.413497 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:55.439446 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.440186 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.444561 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.451143 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.468250 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.471025 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.479379 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.480428 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:55.483848 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:55.485514 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.486354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.491842 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.509652 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:55.518621 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:55.526021 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:55.526463 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:55.596211 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:55.596330 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:55.596975 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.597579 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.601498 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.603729 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.614917 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:55.639549 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:55.645859 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:55.668856 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.682940 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.684967 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.685992 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.707095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.712463 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.728744 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.729908 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.734059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.742257 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.771831 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.780071 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.836369 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.840641 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.857009 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.858412 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.868569 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:55.871510 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.918659 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.925470 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.926760 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.932376 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:55.933329 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:55.946581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:55.961185 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:55.968312 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:55.976969 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 14:03:55.979750 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:55.987701 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:55.996706 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:55.997318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:56.000888 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.007244 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.036976 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.041270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.045279 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.055016 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.060482 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.064527 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:56.077698 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:56.159552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:56.164733 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.170929 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.172396 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.172808 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.189347 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.196113 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.197553 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.211200 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.217586 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.227716 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.234462 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.235069 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.239794 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.243894 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.254511 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.262617 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.264984 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:56.270874 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:56.286154 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.299219 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.316758 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.323534 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.337877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.343076 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.352583 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:56.364784 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.451273 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.460202 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.469750 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.474210 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:56.476410 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.476691 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.479220 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.480450 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.483884 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:56.485527 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.498658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.500609 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.529085 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.552759 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:56.567528 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:56.574759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.586191 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.586225 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.597065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.612100 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.622947 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.624437 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.625172 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.626654 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.628633 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.647608 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.649552 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:56.650555 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:56.750234 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.784883 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.793692 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.796780 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.798233 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.809144 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:56.809417 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.810582 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.821244 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.822849 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.823221 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.823685 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.825403 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:56.850553 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.850558 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:56.862880 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:56.878369 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:56.889366 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:56.890161 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:56.893404 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.902050 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:56.905246 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.907982 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:56.912262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.939414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:56.942205 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.957644 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:56.967835 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:56.968263 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:57.046432 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.048422 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.052999 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.053189 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.063261 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.064675 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:57.076256 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.081598 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:57.084339 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 14:03:57.091638 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:57.110662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.113010 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.114308 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.121790 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:57.133574 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.136631 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.146090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.203668 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.204316 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.204656 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.219125 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.227438 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.230183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:57.245220 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 14:03:57.262030 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:57.313470 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.319618 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.321246 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.321847 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.326258 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.340413 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.379894 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:57.381824 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:03:57.390267 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.390904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.391568 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.391978 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.414338 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.420537 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.424178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.425837 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.431180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.436138 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:57.437373 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.452229 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:57.504112 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:57.544832 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:57.563450 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.565846 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.569504 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.580101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.587558 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.607420 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.610061 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.622380 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.623924 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.628875 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.632944 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.633044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:57.677147 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.680889 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:57.681536 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:57.687101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.703219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.718317 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.737394 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.743739 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.744175 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.757231 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.761309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.767381 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.782253 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.787185 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.790573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.806484 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.809310 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:57.810770 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:57.822885 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:03:57.823908 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:57.829533 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.831739 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:57.843716 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:57.890336 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:57.908396 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:57.924422 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:57.929759 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:57.950592 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:57.960491 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:57.965258 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:57.973865 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:57.992298 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:58.027529 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.035351 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.047073 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.055202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.067659 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.069087 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.076350 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.077520 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.092070 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.098753 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.108036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.108803 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:58.121340 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:03:58.123400 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20240502 14:03:58.127084 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.139281 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.143062 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.143172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.147962 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.155016 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.165730 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.168905 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.169368 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:58.170023 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:58.173457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.191892 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.208632 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:58.235033 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:58.299268 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.310767 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.312214 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.313915 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:58.321126 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:03:58.340260 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.343962 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:58.358810 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:03:58.387599 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:03:58.389096 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:58.413877 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.414996 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.424124 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.430609 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.432876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.437209 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.458276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.459167 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.461825 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.478350 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.482337 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.485491 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.489923 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.506227 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.517833 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.525864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.534605 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.537870 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:58.582536 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:58.630230 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:58.652906 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.659085 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.675498 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.678493 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.691804 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.692667 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.694126 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.701097 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:58.708971 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:03:58.709654 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:58.719023 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.725411 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.732122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:58.739318 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:58.742354 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:58.746886 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:58.747685 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.756318 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.758324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:58.758596 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.759877 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.761301 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.787972 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.796453 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.810227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:58.820534 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:03:58.824151 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.832108 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.833480 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.848699 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:58.851609 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:58.927544 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:58.929540 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:58.932893 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:58.933993 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:58.934051 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:58.945529 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.012125 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.012740 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.013252 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.016770 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.026510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.028302 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.078702 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.080950 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.088996 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.105026 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.125402 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.135972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.140223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.141225 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.141438 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:03:59.151350 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:59.157819 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.164420 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:59.174528 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:59.189144 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:03:59.190164 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.213963 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.227423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.243824 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.244035 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.253613 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.275015 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.279011 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.289147 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:59.289896 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:03:59.290755 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.318617 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.318990 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.337139 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.381140 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.387851 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.394184 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.395051 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:59.415046 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:59.418193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.441946 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.458451 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.462924 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.467640 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.474244 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.476162 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.480026 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:03:59.482193 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:59.489585 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:59.489812 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.490741 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.504204 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.509647 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.516956 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.531294 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:03:59.571786 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:59.571811 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:59.579638 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:59.581094 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:03:59.586313 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.594518 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.596843 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.614650 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.632110 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:03:59.671761 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:59.689687 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:03:59.691244 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 14:03:59.691637 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.694940 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.696409 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.709672 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.728343 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.736917 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.738762 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:03:59.740326 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:03:59.745889 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:03:59.747442 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:03:59.777906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.779249 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.787137 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.787870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.813550 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.823683 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.824697 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.831784 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:03:59.849829 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.856506 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:03:59.860317 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:03:59.870198 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:03:59.871838 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.889060 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:03:59.889480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:03:59.893174 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.012176 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.017153 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.017532 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.020093 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.024827 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.025174 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.026079 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.036707 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.037330 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.040136 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.044461 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:00.045933 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:04:00.048434 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.122581 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.131526 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.137049 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.138444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.139858 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.147491 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:00.178586 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:00.201287 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:04:00.236797 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.242175 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.244961 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.249972 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.256722 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.272354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.285813 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.298162 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.300089 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:00.311350 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:00.314028 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.314260 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.321084 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:00.352475 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:00.365132 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.369701 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.374089 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.377458 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.378329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.380259 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:00.422832 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:00.432008 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:04:00.435364 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:04:00.439831 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.444231 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.444690 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.448350 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.468432 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.473251 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:00.500332 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:00.529600 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.531142 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.535404 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.545727 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.548461 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.551242 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.553411 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.554251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.554636 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.555774 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.556021 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.571774 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:00.637634 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 14:04:00.638480 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:00.648626 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:00.693969 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.700896 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.702766 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.705261 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.723727 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.729743 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.730955 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:00.732758 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:00.733069 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.758450 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.760506 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.773049 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.782615 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.849138 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.852025 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.859161 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.860298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.864742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.867070 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.872771 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.889829 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:00.899153 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.913038 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.915792 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.929481 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.930789 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:00.955806 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:00.956442 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:00.956485 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:00.956713 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:00.992305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:01.028208 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.065235 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.066298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.072417 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.075335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.082808 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:01.092924 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:01.094590 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:01.104162 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:01.109506 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:01.161273 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.174952 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.182204 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.185945 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.186208 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.191929 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.202358 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.205996 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.208557 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.225654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.228780 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.232600 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.233256 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.235689 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.246865 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.249701 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.265115 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:01.270009 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.271553 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:01.285454 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:01.325354 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:01.332479 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:01.403576 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.408917 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.417806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.425241 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.430027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.434789 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.444758 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.456466 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.457121 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:01.481964 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.482467 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.491204 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.492851 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.536170 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.546490 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.549579 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.550318 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.555171 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.571735 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.576023 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.586931 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.587538 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.602690 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.609210 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.624847 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:01.629391 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:01.634570 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:01.680680 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:01.721626 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.722898 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.724829 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.732358 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.736721 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.737159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:01.747715 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.748018 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.754225 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.757036 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.768060 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.786067 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:01.788424 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.797372 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.804914 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.805893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:01.812876 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.818032 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.833266 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:01.886188 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:01.901427 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.920166 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:01.931052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:01.934763 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:01.936648 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:01.950304 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:01.964160 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:01.974524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.030481 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.040726 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.044545 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.044955 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.048770 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.054738 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.069201 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.078265 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.081929 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.089205 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.092720 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.093876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.099426 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:02.104418 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:02.122876 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:02.123373 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:02.129129 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.139582 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.145064 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.147462 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:02.154042 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.233348 25318 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:02.233490 25318 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:02.234063 25318 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:04:02.234542 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:04:02.234696 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:04:02.237001 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:02.237109 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:02.237147 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:02.237397 25318 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:02.246718 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.251360 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:02.254531 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.257372 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.270627 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.280946 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.285759 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.308691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.309906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.313027 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.321101 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.330328 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.340158 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.350327 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.366772 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.377229 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.378325 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:02.383427 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 87 similar messages]
I20240502 14:04:02.385800 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.386590 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:02.388232 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 14:04:02.395316 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:02.433319 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.455504 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.455786 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.456375 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.457975 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.462100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.496809 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:02.512984 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:02.547549 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.557147 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.564728 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.572268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.575553 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.576484 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.576999 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.586724 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.591548 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.596405 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.610114 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.631901 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.639200 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.641896 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.644686 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:02.656723 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:02.662523 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.662935 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.670469 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.675411 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:02.703217 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 14:04:02.716786 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:02.762868 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.772816 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.775158 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:02.782294 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:02.783077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.793651 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.793896 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:02.826201 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:04:02.826835 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:02.831921 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.841264 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.852749 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:02.856904 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:02.864368 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.864421 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.865465 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:02.873544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:02.874873 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:02.894037 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:02.894318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:02.910881 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:02.915532 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:02.967378 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:03.026580 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:03.053378 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.054193 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.058861 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.063377 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:03.076097 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:03.087810 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.096829 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.100875 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.101670 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.102643 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.103955 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.104847 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.109200 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.113214 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.113620 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.113874 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:03.125056 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:03.127300 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.129778 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.137773 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.147171 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.149713 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.168193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:03.178427 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:03.183379 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:04:03.191938 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.195230 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.195644 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:03.263813 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:03.329892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.346853 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.347457 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.357482 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.381970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.391958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:03.392057 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:04:03.398660 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.398792 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.399709 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.413472 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.414268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.417440 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.433967 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.435671 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.435916 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.455360 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.465332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.475229 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:03.475571 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 14:04:03.501543 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.514014 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.514699 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.519521 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.535601 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.541217 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.623687 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.626448 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.632089 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.632449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:03.637547 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:03.638312 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.658182 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.682614 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.685937 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:03.692575 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:03.710175 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:03.712940 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:04:03.725759 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.727855 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.733649 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.736371 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.737087 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.738533 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.740028 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:03.755401 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:03.769701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:03.779224 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.788132 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.791620 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:03.792307 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:03.804885 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:03.810585 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:03.815611 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.815663 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.815936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:03.874640 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:03.905860 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:03.925356 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.928666 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:03.941421 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:03.945945 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:03.961429 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:03.966312 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:03.975189 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:03.988747 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:03.989010 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:03.989080 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:03.991751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.001333 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.002697 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.008249 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.023151 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.032789 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.041009 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.041296 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.045189 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.053329 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.057876 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:04.141638 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:04.174477 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.182173 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.187837 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.188541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.198143 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.213205 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.215572 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.215902 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.219089 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.224778 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.235152 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.246130 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.257211 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.268335 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.277966 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.279794 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.282333 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.323016 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:04.336184 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:04.336980 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:04.415206 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.427971 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.443493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.451609 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.451714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:04.454026 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.454448 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:04.462656 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.463403 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.463512 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.476464 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.476986 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.497754 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.516742 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.521996 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.534144 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.540665 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.543597 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.545223 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.548218 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.549412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.553248 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.584201 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.585834 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.589110 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.593233 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:04.606034 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:04.686169 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.713963 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.733767 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.735556 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:04.737861 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 14:04:04.744189 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:04.745241 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.746640 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.748114 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.760035 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.760251 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.787197 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.793828 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.805388 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.805832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:04.808145 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.809105 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.809363 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:04.826213 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 14:04:04.839367 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:04.840374 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:04.840591 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:04.865918 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:04.878646 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.884364 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:04.889176 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:04.891944 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:04.899960 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.907590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:04.908185 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:04.923764 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:04.932080 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:04.994673 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:05.004257 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.011575 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.014173 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.021415 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.031621 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.035796 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.086453 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.091629 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.092806 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.092800 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:05.096436 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:05.104720 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.106822 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.110128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.113509 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.117130 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.123776 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.152915 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.154621 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:05.168375 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:05.214612 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.217898 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.220185 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:05.224611 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:05.225056 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.251132 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.259383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:05.260313 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:05.307758 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:05.310722 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:05.325939 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.333338 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.343024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.345238 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.363710 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.382629 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.384904 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.385316 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:05.394824 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:05.395275 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.398953 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.407205 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.409489 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.411096 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.412442 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.417192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.441406 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.457283 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:05.505926 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:05.508642 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:05.513581 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:05.520387 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.521185 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.524173 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.524252 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.527623 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.594336 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.605825 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.611052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.612895 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.614041 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:05.621995 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:05.625435 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.631552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.639364 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.645416 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.656419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.669409 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.683207 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:05.704234 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:05.734119 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:05.768649 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.769403 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.781345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.784723 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.785151 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.796234 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.800562 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.805158 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.822453 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.834594 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.842072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.842334 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:05.856132 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:05.893020 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:05.906380 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:05.917260 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.922086 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.926402 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:05.928223 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.930389 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.969527 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:05.970026 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:05.981316 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:05.982308 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:05.982374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.001080 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.084514 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.090560 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.090636 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:06.103554 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:06.113723 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.116653 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:06.126299 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.129838 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.151135 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.185046 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.198814 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.205921 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.212445 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.219784 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.222643 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:06.232437 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.233958 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.238871 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.242841 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.243727 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.253716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.257010 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:06.259250 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:06.262707 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.264241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.265313 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.265895 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.294950 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:06.296176 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.346545 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.354445 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.381088 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:06.386301 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.388804 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.392307 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.403740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:06.421120 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:06.470780 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:06.485646 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.500994 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.502738 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.520037 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.520287 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.528944 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.531183 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:06.544521 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:06.544575 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.544827 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.545293 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.552096 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.557022 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.572422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.574113 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.576490 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.584501 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.593701 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.608507 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:06.655272 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.686827 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.694069 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.701460 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.704479 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.705914 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.723609 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.755321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.759894 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.766661 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.766777 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:06.770881 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:06.789707 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:06.791102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.824542 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.826536 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.827083 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.831681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.839846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.852099 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:06.864135 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:06.919162 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:06.924371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:06.930519 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:06.933216 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:06.935124 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:06.938078 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:06.974756 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:06.995021 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.003618 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.006688 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.009426 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.015456 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.019006 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.094821 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.098801 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.111785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.112179 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.113130 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.121502 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.136298 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.137637 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.141021 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:07.142578 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:04:07.153412 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.166929 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.169863 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:07.173614 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.178710 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:07.179126 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:04:07.205144 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.241263 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.246909 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.246995 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:07.256330 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.262885 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.274729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:07.274840 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.289583 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.305477 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.306823 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:07.319278 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:07.321087 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:07.330420 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.344570 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.346325 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.352807 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.353225 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.360920 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.376556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.383596 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.392772 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.403157 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:07.413599 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:07.418644 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:04:07.423074 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 14:04:07.426903 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.431262 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.432667 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.433615 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.440132 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.442922 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.525614 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.528163 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.530606 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.531200 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.534953 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.538771 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.592664 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.592888 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.593034 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.602550 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.606105 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.607689 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:07.654481 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:04:07.694061 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:07.702749 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.703558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.706940 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.710584 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.723661 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.737098 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.764186 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:07.770963 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.774294 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:07.781634 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:07.785019 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.789324 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.790066 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.828936 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.836680 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:07.836825 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:07.836825 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:07.839426 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.848251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:07.850224 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.860476 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:07.863262 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:07.867130 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:07.894037 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:04:07.932904 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:07.938824 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:07.940376 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:07.945325 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:07.945397 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:07.960153 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:07.980984 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.027827 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.029112 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.042932 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.044951 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.046046 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.048331 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.049813 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.050927 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.060853 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.069682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:08.073043 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 14:04:08.079015 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:08.085274 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.086956 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.092461 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.098263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.106266 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.124781 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.136209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.148985 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.197933 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.202416 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.211143 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.212471 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.220875 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.246068 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:08.288224 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:08.293987 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.304684 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.305963 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.310827 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.320886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.321662 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.333413 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:08.341593 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:08.350704 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.353087 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.353356 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.360728 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.386840 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:08.392060 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:04:08.432163 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:08.507382 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.511132 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.530745 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:08.539932 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:04:08.540047 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:08.544823 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:04:08.556800 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.563728 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.564805 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.568079 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.568564 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.575821 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.579614 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.584513 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.585162 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.593400 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.600495 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.626559 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.627512 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.631018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.633039 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.636260 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.641510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.643083 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.653179 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:08.701000 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:08.702924 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:08.743613 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:04:08.744372 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:08.764325 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:08.787211 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:08.797394 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.802832 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:08.814239 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:08.832751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.834283 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.840137 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.845034 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:08.845294 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.846894 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.863703 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.867363 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.875097 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:08.876879 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:08.878388 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:08.900142 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:08.931640 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:08.936242 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:08.937100 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.943199 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:08.953194 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:08.961161 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:08.972599 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.977506 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:08.978049 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:08.992455 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:08.999240 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:08.999675 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.010949 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.018963 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.026631 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.117997 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.118227 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.125852 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.133823 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.136122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:09.139024 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:09.159354 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.178586 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.179090 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.181660 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.199806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.209517 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.209539 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.234548 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.234925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.236527 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.237903 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.261642 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.262574 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.281452 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.289194 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.289815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.295898 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.315832 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.330066 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:09.387960 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:09.440987 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:09.453155 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:09.461041 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.461777 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.481899 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.490422 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.496022 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.509838 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.511329 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.521337 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:09.521620 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:09.524580 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.526301 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.532298 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.570289 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:09.580933 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:09.610734 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.615295 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:09.629628 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:09.639307 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.643224 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.650187 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:09.672022 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:09.683995 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.689011 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.691490 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:09.695144 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:09.697495 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.702723 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.717056 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.724342 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.769006 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.769811 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.772202 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.788712 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.796607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.798713 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.809798 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.815114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:09.816586 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.831696 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.833986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:09.846786 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:09.877816 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 14:04:09.905982 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:09.949930 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:09.950022 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:09.962734 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:09.964208 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:09.974964 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:09.992100 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:09.994282 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 14:04:09.994694 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.004918 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.005735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.006027 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.008976 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.014804 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.017257 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.043043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.064352 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:10.070489 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:10.071658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.082509 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.096446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.103597 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:10.104455 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:10.186373 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.197806 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.204929 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.214663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.215332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.216738 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.218516 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.232544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.241113 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.257074 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:10.259001 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:10.274667 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.277585 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.278877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.299479 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.301306 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.318344 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.319319 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.326049 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.331477 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.336426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:10.339349 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:10.339480 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.341296 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.343122 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.367977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:10.375298 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:10.487901 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:10.491042 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.496781 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:10.505811 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:10.507400 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:10.508258 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:10.515551 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:10.518045 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:10.527316 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:10.540551 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.541859 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.546705 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.551532 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.574331 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:10.606765 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:10.606848 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.616736 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.617270 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.619130 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.626367 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.627144 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.635499 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.637697 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.646245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.653182 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.653712 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.660439 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.676577 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.678454 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.685736 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.689280 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.692422 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.700003 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.701568 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:10.727303 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:10.732146 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.737681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.806093 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.812940 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.815424 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.816203 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.816797 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.835410 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.883689 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.883977 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.886319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:10.894780 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.896111 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:10.897760 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:10.912328 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.948113 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:10.950768 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:10.953841 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:10.958071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:10.959116 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:10.976904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:11.038854 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:11.051450 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:11.051563 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.053475 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.053891 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.064409 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.077036 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.079649 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.117663 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.120011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.126437 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.128360 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.129766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.130343 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.214375 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.229171 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.229908 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.234267 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.237001 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.259379 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:11.304841 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:11.305073 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.305742 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.308650 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:11.311084 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.313968 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.331818 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:11.343449 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.345307 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.350446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:11.378504 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:11.394057 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.399845 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:11.406648 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.409162 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.425789 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.427493 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.432893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.438390 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.446427 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.453850 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.462852 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.464888 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.478154 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.502727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:11.525597 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:11.535780 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.544680 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.547961 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.577762 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.578430 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.580500 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.595237 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:11.620697 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:11.623934 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:11.625231 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.635550 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.642627 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.644887 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.645104 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.655141 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.655758 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.657274 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.669773 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.672287 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.700513 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.732102 31471 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:11.732226 31471 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:11.732743 31471 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:04:11.732987 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:04:11.733116 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:04:11.734100 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:11.734220 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:11.734270 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:11.734412 31471 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:04:11.814082 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.820312 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.828511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.833328 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:11.844801 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.848079 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.854202 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:11.854954 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.860688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.878330 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:11.881337 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.907918 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:11.919195 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.925334 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:11.932559 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:11.948899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:11.952795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:11.973908 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:11.983827 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:11.985044 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:11.997416 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.014923 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.030522 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.033247 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.033474 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.034113 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.038573 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.039031 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.044142 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:12.062044 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:12.072410 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.077543 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.085247 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.097651 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.100658 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:12.116794 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:12.158044 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.158668 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.159993 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.160053 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.167022 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.186731 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.189560 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:12.225999 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:12.237918 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.247802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.251062 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.252169 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.254215 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.272017 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:12.275753 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:12.291976 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.316437 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.324956 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.326903 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.334112 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.339896 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.352643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.382542 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.384387 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.384938 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.390889 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.409833 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.416471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:12.419131 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:04:12.437100 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.443292 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.453279 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.456444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.459384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.459638 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:12.569316 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:12.570241 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.575392 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.583096 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:12.583665 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.587690 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.589745 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:12.594966 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.597321 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.603346 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.614631 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.623937 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.650640 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.652801 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.666477 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.675474 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.680230 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.684541 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.688571 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.690680 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.718959 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.735036 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.742808 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.750936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.759014 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.765609 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.792996 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:12.801542 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:12.826411 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 14:04:12.841351 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.857556 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.869832 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:12.871556 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:12.879900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:12.886745 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:12.888126 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:12.888460 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:12.899472 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:12.907016 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:04:12.910964 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:12.914917 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:12.916373 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:12.917586 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:12.939685 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:12.948877 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:12.957795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:12.985591 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:12.987733 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:12.991794 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:12.994359 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:12.998677 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.002554 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:13.008965 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:13.014891 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.017833 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:13.037142 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 14:04:13.092895 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.095284 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.100100 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.102283 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.115155 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.123533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.164049 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.164834 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.166059 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.167491 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.168566 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.181799 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:13.182729 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:13.186183 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:13.221148 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.222688 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.234464 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.237057 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.262094 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.271956 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.319214 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.319820 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.322510 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.323594 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.327636 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.363602 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.375538 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.376890 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:13.379365 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:13.392504 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240502 14:04:13.411463 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.412621 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.418751 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.435523 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.439545 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.446048 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.447202 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.450908 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.453944 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.488771 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.521001 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.525054 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.528530 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.536724 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.538573 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.546422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:13.610646 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:13.615995 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.616477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.619863 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.627599 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.636503 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.657517 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:13.665315 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:04:13.689168 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.689739 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.698318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.698711 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.701582 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.725054 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.749630 31471 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:13.749755 31471 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:13.750171 31471 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:04:13.750880 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:04:13.751026 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:04:13.751415 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:13.751682 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:13.751735 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:13.751911 31471 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:04:13.769192 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.770787 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.770924 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:13.778149 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:13.781124 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:13.784911 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.788486 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:13.788736 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:13.799669 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:13.804121 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:13.805657 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:04:13.806246 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:13.812496 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.813464 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.816552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.830987 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:13.836992 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:13.840776 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:13.870455 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:13.920107 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:13.933342 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:13.935815 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:13.938444 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:13.944172 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:13.945564 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:13.957012 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:04:13.961061 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:13.994663 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:14.012352 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.015308 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.015755 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.022508 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.024832 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.036463 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:14.041937 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:14.079241 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:14.083397 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:14.096518 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.098785 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.109211 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.109719 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.111340 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.130510 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.155058 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.167366 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.174479 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.177521 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.196910 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.198871 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.208035 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.209838 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.213877 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.236793 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.240473 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.242395 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:14.259210 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:14.335064 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.343248 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.346172 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.348279 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.351562 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.357519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:14.394450 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:14.414516 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.421376 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.429572 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.434194 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.434957 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.443499 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.453528 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.467444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.469624 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.473444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:14.477183 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:14.482393 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.485186 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.547197 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.551877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.554037 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.554047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.554630 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.563964 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.564487 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.570941 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.574896 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.575749 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:14.576941 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:14.583778 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:14.589640 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:14.599228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:14.631114 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:14.638464 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:14.651454 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:14.696125 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.711537 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.716910 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.721515 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.724088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.727741 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.743281 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:14.752678 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:14.760910 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.774932 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.781986 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.783134 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.795140 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.803215 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.807219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.810362 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.837364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.849844 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:14.868713 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:14.942921 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:14.948122 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:14.971032 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:14.971221 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:14.988436 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:14.988513 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:15.000829 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.006283 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.008029 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:15.010134 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:15.011123 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:15.012431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.013468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.016039 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.023108 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.050983 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.053053 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.055979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.057967 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.059837 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.063890 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.139621 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.149145 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.150818 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.159579 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.174059 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:15.200789 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:15.205554 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:15.221714 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:15.225503 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.242962 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.248186 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.249173 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.249435 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.256549 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.266271 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.271621 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.311935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.312718 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.313532 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:15.316877 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:15.325402 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.331694 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.333364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.336645 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.338465 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.358389 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:15.369875 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:15.371188 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:15.386461 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:15.394304 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 14:04:15.416790 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:15.451536 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.465855 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.469780 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.479780 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.483059 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:15.483377 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:15.487571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.543049 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.558384 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.563381 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.569368 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.570410 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.575371 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.592833 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.593256 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.598096 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:15.605823 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:15.617949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:15.620951 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:15.624425 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.648142 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.649425 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.652693 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.654520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.658140 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.676426 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.692104 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:15.692848 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 14:04:15.739765 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:15.753290 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:15.764842 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.772003 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.780237 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.785729 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.794893 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.796388 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.847597 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.848171 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:15.848755 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.855711 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.865343 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.881881 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:15.937614 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.940611 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:15.945796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.947551 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:15.957367 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:15.961076 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.970067 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:15.983685 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.001369 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.002770 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.015457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.022948 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.102290 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.110214 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.126919 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.130734 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.141355 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.145411 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.150327 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.158509 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.163080 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.163318 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.175062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.177358 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:16.189584 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:16.196139 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:16.199677 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.207794 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.209548 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.220057 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.221872 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.232295 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.283196 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.295650 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.297767 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.299667 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.310220 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.317374 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:16.356484 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:16.374639 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:16.378470 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:16.389062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.391702 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.392988 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.398725 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.412431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.413141 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:16.421319 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:16.438979 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:16.446185 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.450430 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.463446 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:16.465962 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:16.467078 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.470427 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.473515 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.483160 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.484395 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.502842 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.517588 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.524518 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.525094 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.614396 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.614993 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.615410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:16.627416 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:16.628324 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:16.630532 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:16.631109 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:16.650995 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:16.658794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.681478 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.687903 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.689569 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.692764 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.693409 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.702987 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.703308 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:16.710536 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:16.711030 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:16.715404 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.732578 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.736588 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.746474 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:16.751461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:16.821170 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:16.867573 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.869927 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.871836 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.885962 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.916684 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:16.917268 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:16.934085 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:16.941151 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:16.958828 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:16.976476 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.976742 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:16.981106 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:16.986011 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:16.992447 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.001561 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.002617 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.006976 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.016343 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.017431 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.020862 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.022047 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.022346 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.023067 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.023831 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.024466 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.075245 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:17.100178 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:17.130606 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:17.143169 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:17.153209 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:17.174687 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:17.234457 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.235020 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.236657 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.238103 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.242933 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.244719 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.249517 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.251650 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.252103 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.255556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.289881 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.302809 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.312094 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.318928 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.319499 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.321460 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.327531 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:17.346064 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:17.347889 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.390787 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.395009 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.398128 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.403065 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.413087 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.414635 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:17.479140 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 14:04:17.486990 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:17.507998 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.508888 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.511169 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.518486 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.531330 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.542618 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.557679 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.559671 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.571398 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.573951 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.574024 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.586390 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:17.599733 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:17.615132 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.623662 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.635628 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:17.639314 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:17.641893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.644804 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:17.655856 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:04:17.656766 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:17.663748 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:17.690757 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:17.706709 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.732583 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.740072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:17.742384 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:17.749702 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.752249 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.765213 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.819926 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.820845 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.821193 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:17.833809 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:17.838673 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.854533 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.859581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.862888 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:17.868714 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:17.871688 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 14:04:17.873003 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:17.874439 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:17.888025 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:17.888123 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:17.901722 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.903226 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:17.904373 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.910235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:17.910744 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:17.942524 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:17.951162 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:04:17.975466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:18.013682 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:18.043875 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.047659 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.058645 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.061609 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.063056 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.083654 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.083901 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:18.096663 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:18.097061 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.097851 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.098497 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.099555 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.107940 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.169257 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.174361 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.175887 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.176669 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:18.178843 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:18.189939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.195025 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.260025 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.266933 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.268910 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.282927 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.296953 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.311769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:18.323953 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:18.337071 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:18.338232 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:18.338996 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.343906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.349169 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.356002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.358700 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:18.409986 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:04:18.410123 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.416612 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.419697 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.430876 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.431753 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.457616 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.510129 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:18.510673 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:18.512117 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.543709 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.547183 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.559963 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.566679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.567282 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.579643 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:18.586198 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:18.594652 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.599807 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.600098 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.619273 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.653391 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.656949 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.662472 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.663115 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.666666 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.669350 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.745386 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.748651 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.751616 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.759511 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:18.774338 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:18.782595 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.783672 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.786908 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:18.788957 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:04:18.789801 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:18.792133 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:18.795310 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:18.801889 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:18.802664 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.812947 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:18.827214 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:18.845255 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:18.855074 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:04:18.856405 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:18.879101 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:18.896484 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:18.905998 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.915155 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:18.921877 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:18.929742 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:18.940671 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:18.945845 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.948913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:18.954149 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:18.977886 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:18.984730 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.988610 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:18.994721 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.006848 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.016433 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.038019 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.055253 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.062407 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.062867 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.063098 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.080582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:19.110996 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:19.132146 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.141957 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.143898 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.146601 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.148131 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:19.152864 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:04:19.153420 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:19.154338 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:19.187244 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:19.203210 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.203477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.204468 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.206868 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.207768 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:19.221158 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:19.224645 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.292544 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.295419 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.305105 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.318863 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.321949 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:19.330258 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:19.359584 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.384706 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.391391 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.393299 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.400905 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.414680 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.421234 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.466369 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.472478 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.477380 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.481699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:19.488166 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:19.495921 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.503413 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.509972 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:19.519942 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:19.520802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.523969 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.527350 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.528340 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.530673 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:19.591096 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:19.617005 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:19.623657 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:19.628643 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.634701 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.635193 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.651859 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.672423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.695633 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.706164 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.709954 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:19.710213 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:19.711174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.716897 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.724074 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.730974 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.737931 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.744539 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.744969 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.751458 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.774366 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.828263 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.839666 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.845088 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.857371 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.857926 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.863126 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.927090 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:19.930431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.935041 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:19.939599 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.941340 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.945231 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.962280 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:19.968859 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:19.981490 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:19.983021 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:19.989735 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:19.991382 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:19.995798 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:20.045140 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:20.065426 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:20.066309 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:20.095381 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:20.101109 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:20.102408 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.109426 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.109537 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.125519 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.133615 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.136999 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.140447 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.155138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.161715 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.162971 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.169016 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.174294 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.178932 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.192965 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.195420 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.201534 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:20.204794 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:20.223959 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:20.318323 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:20.346545 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.348299 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.351655 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.355069 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.358898 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:20.365252 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 14:04:20.366714 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:20.369074 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:20.369516 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.371266 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.372006 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.376798 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.412657 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.412973 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.419354 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.450302 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:20.461038 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:20.463987 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.473937 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.479249 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.486905 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.488201 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.497850 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.501181 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.503806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.515070 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.532624 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.547240 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:20.628006 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:20.630879 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.631842 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.641629 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:20.656350 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:20.657682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.660401 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.668622 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.699368 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.704016 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.705241 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.721755 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.725960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.731068 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:20.754007 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:20.756534 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.759459 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.766283 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.779908 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.780084 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:20.782500 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:20.794051 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 14:04:20.796547 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.805855 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.813498 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.830677 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:20.844534 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.861546 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:20.876102 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:20.918370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:20.923607 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:20.934216 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:20.945621 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:20.947043 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:20.948590 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:20.982268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.007210 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.008100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.014482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.015002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.020239 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.035202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:21.041201 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:21.072003 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.082867 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.086288 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.094033 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.094465 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.099134 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.102699 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.106164 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.119285 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.125859 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.132651 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.134708 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.218446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.223245 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.225684 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:21.225739 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.226928 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.228435 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.238088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:21.246372 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.283900 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.284461 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.293753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.305014 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.311182 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.336567 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:21.384124 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.399410 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.400220 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:21.402243 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:21.404323 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.408607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.409245 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.416003 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:21.450757 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:21.471474 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:21.482774 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.482820 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.489373 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.490958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.498895 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.501336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.503373 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.515280 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.518566 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:21.527122 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:21.527834 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:21.537833 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:21.548708 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.554378 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.555732 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:21.570894 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:21.570998 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.609032 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.610250 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.618155 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.619925 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.629073 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.635362 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:21.653856 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:21.653945 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:21.701812 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:21.728415 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.740659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.743393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.746811 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.747812 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.761058 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.764688 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.768399 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.768981 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.769675 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.773510 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.780458 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.792955 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.799121 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.804256 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:21.816695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.835834 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.857148 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:21.868604 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:21.907653 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:21.955026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:21.955060 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:21.961164 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:21.961629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:21.962316 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:21.963692 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.017223 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.025991 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.032409 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.032986 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.055830 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.056305 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.079716 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.089980 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.108021 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.114295 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.115664 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.124354 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.133476 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.137399 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.164357 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.164573 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.168459 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.177788 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:22.188248 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:22.210692 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:22.273103 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.275568 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.288627 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.300230 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.313460 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.317714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.323379 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.332340 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.333667 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:22.334082 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:22.354100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.366727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.371155 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.372984 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.377154 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.390147 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:22.402175 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:22.405592 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.405864 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.421416 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:22.444128 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:22.451522 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:22.461972 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.472018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.472843 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.477115 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.491032 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.503065 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.550451 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.559783 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:22.572513 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:22.577615 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.579280 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.579974 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.598882 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.612612 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.614603 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:22.617871 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:22.624727 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.629761 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:22.637646 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:22.648255 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:22.651298 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:22.652664 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.677994 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.681977 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.683928 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.686014 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.689090 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.689457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:22.697268 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:22.744990 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:22.809190 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.809724 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.811085 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.815281 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.828545 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.857753 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:22.862299 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:04:22.863021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.872954 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.875857 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.882639 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:22.885761 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:22.891463 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:22.892563 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.923918 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.941776 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:22.951568 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:22.952967 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:22.959472 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:22.966683 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.969599 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:22.976047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:22.978828 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.979498 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:22.988878 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:22.991089 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:23.010329 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:04:23.014168 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.015089 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.112766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.117274 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.119252 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.120643 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.133638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.144526 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.155360 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.156606 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.174193 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.179765 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.189013 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.204855 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.205310 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.206971 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.217013 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:23.218833 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:23.224893 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.240854 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.242472 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.355899 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.362779 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.363548 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.367516 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.371951 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.376827 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.379953 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.390861 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.400296 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.419296 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:23.432524 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:04:23.433699 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:23.438506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.449328 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.452945 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.461459 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.470335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:23.487010 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 14:04:23.489301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.497460 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.498080 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.502557 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.507105 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.508173 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.520027 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:23.538115 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:23.546773 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.549890 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:23.556392 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 14:04:23.618959 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.629725 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.631932 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.643674 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.649907 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.650835 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.651947 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:23.659529 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:23.659824 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:23.692071 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.693174 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.706169 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.706420 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.723107 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.784849 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.788761 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.800762 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.812774 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:23.814883 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:23.821604 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:23.835886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.843225 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.844161 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.846105 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.847649 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.863116 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:23.864172 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:04:23.879185 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:23.882514 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:23.896183 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.945135 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.946734 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:23.948776 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:23.950518 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:23.952523 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:23.955361 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:23.966676 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.967262 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:23.968364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:23.981151 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.000514 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.018362 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.019327 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:24.045735 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:04:24.046526 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:24.063876 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.067646 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.068794 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.071800 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:24.078096 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.079123 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.080158 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:24.089339 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:24.094604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:24.111438 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:24.147197 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.184165 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.191066 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.200246 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.201964 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.204674 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.208770 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.212297 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.219305 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.220687 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:24.251269 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:24.251977 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.258188 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.260345 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:24.308533 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:24.331614 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.342927 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.358228 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.358808 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.366467 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.383015 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.400373 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.408308 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.410017 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.414639 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.419525 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.421315 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.432771 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.439225 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.464630 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.470552 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.476447 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.482074 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.495249 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.524423 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:24.535831 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:04:24.536453 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:24.544243 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.546231 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.546820 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.550846 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.563741 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.568538 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.569075 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.574299 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:24.585768 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:24.585870 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.604933 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.614856 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.617327 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:24.645552 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:24.666726 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.669453 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.673393 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.673424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.673853 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.676445 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:24.697445 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.731988 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.738102 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.741154 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.753069 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.755390 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.764753 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:24.771539 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:24.816259 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:24.817479 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:24.818403 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.822463 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.824452 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.841019 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:24.898188 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:24.899422 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:24.899474 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:24.910748 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:24.924921 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:24.927109 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:24.927362 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:24.931291 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:24.999658 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:25.011601 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.028311 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:25.038174 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:25.041491 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.043819 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:25.048235 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:25.051481 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.056918 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:25.062064 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:25.066052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.067013 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.068253 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.069276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.082137 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.099870 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.100454 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.105584 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.106407 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.111492 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.134634 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.150913 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:25.151643 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:25.151763 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:25.211305 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:25.230757 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.234362 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.243108 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.251257 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.256778 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.266717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.287761 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.313706 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.330559 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.330858 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.337267 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:25.346985 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:25.352121 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.355945 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.361022 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.373800 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.389370 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.395278 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.396399 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.416157 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.420478 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.428079 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.428669 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.429175 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.429441 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:25.441464 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:25.506070 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.515940 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.516518 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.525481 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.544796 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.546624 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.599356 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.599454 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.606221 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.622148 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.622733 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.625025 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.627447 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.650620 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.657541 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.658150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.666855 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.673758 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:25.751979 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:25.806479 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.820138 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.825781 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.827037 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.829573 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.836988 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.851351 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.851982 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.855902 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.856457 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.864706 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.878403 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:25.886550 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:25.903785 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:25.906599 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:25.911026 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:25.933779 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.935241 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:25.944185 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:25.961258 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:25.962939 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:25.973012 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.985018 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:25.995440 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.013424 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.015157 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:26.047086 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 14:04:26.051296 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 14:04:26.081012 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.099416 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.111711 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.115876 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.120420 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.128746 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.136873 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.155894 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:26.156389 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:26.160763 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.164728 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.165869 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.171545 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.190587 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.196588 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.199995 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.208395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.212193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.226011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.226481 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:26.248813 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:26.287276 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.304381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.319478 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.322207 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.324679 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.325548 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:26.330546 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.352540 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.365471 8123 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:26.365617 8123 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:26.366202 8123 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:04:26.366925 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:04:26.367060 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:04:26.367188 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:26.367338 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:26.367408 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:26.368000 8123 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:04:26.372815 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.386808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.390692 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.395556 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.405310 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.417939 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.423907 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:26.426559 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:26.427656 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.429364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.437662 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.450078 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.451586 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.481448 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:26.493268 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:26.506989 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.512920 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.521924 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.523326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.527868 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.528970 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.530766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.604166 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.607846 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.608662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.614277 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.617727 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.628100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:26.658461 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:26.662024 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.670749 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.671329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.672859 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.690877 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.703222 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:26.750384 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:04:26.750747 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:04:26.762977 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.790647 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.800719 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.803027 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:26.803851 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:26.811534 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.818140 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.832892 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.841275 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.848459 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.871359 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:26.872088 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:26.876183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.877241 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.884333 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.888801 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.895648 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.901242 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:26.903992 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:26.908048 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.912822 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.942849 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:26.943308 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 14:04:26.944197 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:26.946683 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:26.950210 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:26.961208 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:26.972450 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:26.983518 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:27.028726 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:27.065682 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.071075 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.071748 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.072208 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.097714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.117524 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.118011 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.136453 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:27.144752 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.147289 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.148906 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.158533 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.160276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.167721 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.177040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.180300 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:27.189229 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:27.200325 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:27.209394 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.209720 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.212100 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:27.300474 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:27.319164 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.332746 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.332746 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.349947 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.354560 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:27.361539 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:27.363517 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.390655 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.408105 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.409785 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.410879 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.415289 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.418221 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.431260 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.449379 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.452452 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.452896 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:27.461405 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.466363 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.478677 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:27.489753 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.500026 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.501272 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.508364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.515177 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.518608 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.528849 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:27.534586 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:27.550141 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:27.601830 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.604686 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.608605 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.612296 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.614490 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.634604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:27.640947 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:27.664052 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.685365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.692222 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.696683 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.698212 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.701098 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.715420 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.731168 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.740892 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.741187 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.746277 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.754314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.754571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:27.772519 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:27.796649 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.796824 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.809468 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.820643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.824492 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.826097 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.897048 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.897301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.911135 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:27.916385 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.917258 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.924541 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:27.937672 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:04:27.957444 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:27.959882 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:27.965574 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:27.975402 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:27.977495 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:27.979849 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:27.987010 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:28.015487 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:28.026718 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 74 similar messages]
I20240502 14:04:28.032606 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.037317 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.045666 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.049912 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.055581 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.058420 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.114269 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.117534 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.121016 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.123046 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.127939 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.129622 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:28.129727 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:28.195537 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.202710 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.206554 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.226956 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.227916 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.231237 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.241469 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.250275 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.252679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.261225 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.268409 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.285688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:28.308337 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.369164 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.371775 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.378332 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.384052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.386420 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.399448 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.457587 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:28.464026 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:28.468238 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:28.486594 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.490916 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.491228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.494498 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:28.495085 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:28.498301 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.498896 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.503264 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.506793 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.524180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.526943 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.532701 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.537031 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.538106 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:28.539969 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:28.548729 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:28.551061 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:28.554788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.567433 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:28.572069 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 14:04:28.583601 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.685911 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.685938 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:28.687170 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.696316 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.701436 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.711869 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.715119 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.717352 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.732718 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.738380 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:28.741789 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.745708 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.751914 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:28.752646 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.805048 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.811544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.812328 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.812700 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.812956 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:28.828190 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.855938 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:28.858363 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.914966 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.926393 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:28.931778 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:28.932240 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.935380 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:28.939893 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:28.946768 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:28.987861 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:28.991493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:28.999770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.000795 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.002283 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.003379 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.022650 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.026831 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.035090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.036825 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.039741 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:29.046182 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:29.056867 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:04:29.067925 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:29.069916 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:29.079771 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:29.121829 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.122480 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.126019 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.126466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.137948 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:29.149711 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:29.154619 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:29.239780 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:29.247397 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.248265 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.256752 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:29.259836 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:29.273088 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.289239 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.295635 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.311990 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.322019 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.323047 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.324297 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.327086 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.333415 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:29.362900 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:04:29.369815 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.373214 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.388799 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.391217 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.392357 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.393102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.425976 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.429138 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.429138 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.445082 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:29.458930 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:29.461333 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.464388 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.473098 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.473781 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.474323 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.482909 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.496897 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.498245 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:29.514889 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:29.516947 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:29.535893 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:04:29.536581 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:29.553959 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:29.572443 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:29.601603 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.606384 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.606410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.611636 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.624455 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.627190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.631453 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.631793 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.639220 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:29.639312 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:29.641741 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:29.644042 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.667871 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.686094 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.726497 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.729730 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.730643 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.732465 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.745201 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.773983 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.777222 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.781103 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.782871 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.793915 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.800112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.815546 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.884024 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.886299 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.890501 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.890779 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:29.891544 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:29.909276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:29.968107 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:29.974125 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:29.982398 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:29.982417 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:29.985929 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:29.991853 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:29.996845 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:29.997164 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:29.999454 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:30.030689 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:30.037946 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:30.045068 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:30.048967 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:30.076661 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.088104 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.093850 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.108832 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.108826 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.114313 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.116233 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.123633 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.125053 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.125805 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.130231 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.161520 29272 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.161870 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.161520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.176951 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.183823 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.197074 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.197145 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.257544 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.258752 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.275168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.282373 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:30.284037 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:30.286926 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.289237 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.362442 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.363116 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.365602 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.371255 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.371562 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.371980 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:30.388082 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:30.416957 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:30.419345 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.423542 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.424998 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.426772 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.442687 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.444607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.444656 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.463881 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.465276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.465663 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.467797 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:30.496490 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:30.501180 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.577322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.579480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.582190 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:30.592557 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:30.593137 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.615559 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.623574 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.656248 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.663507 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.664054 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.664566 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.672618 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.687750 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.690260 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.691746 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.704666 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.716138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.731920 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.740435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.770586 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.776172 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.777024 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.788470 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.791973 9253 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:30.792073 9253 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:30.792609 9253 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:04:30.794216 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:30.794292 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:30.794319 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:30.794481 9253 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:04:30.794888 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:04:30.795032 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
I20240502 14:04:30.807595 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.809317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:30.877624 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:30.887961 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:30.891314 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:30.896852 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.925376 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:30.926514 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:30.926856 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.938639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:30.939132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.941112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:30.954202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:30.965451 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.984570 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:30.984622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:30.993695 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 14:04:31.015970 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:31.018319 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.024323 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.034538 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.036126 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.036742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.038267 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:31.064921 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:31.081838 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 14:04:31.089118 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:31.098975 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.102174 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.105772 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.108654 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.114449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.144479 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:31.198036 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.202548 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.205682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.207027 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.210639 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.211645 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.212152 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.222865 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.236639 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.246220 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.260167 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.260308 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.276796 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.289891 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.294037 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.299026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.314901 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:31.321429 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.323645 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.329098 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:31.406273 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:31.416913 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.422730 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.424764 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.429692 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.433717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.452837 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:31.460783 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:31.463508 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.466992 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.468526 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.472047 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.474090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.487550 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.508769 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.513960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.518478 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:31.525617 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:31.542927 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.544406 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:31.544778 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.548585 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.565461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.615572 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.617269 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.620466 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.626545 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.637794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.647171 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.695907 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.696096 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.704488 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.717311 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.732081 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.733222 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.743669 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.744928 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.747216 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.751195 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.756701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:31.757529 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:31.763851 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:31.775393 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:31.788218 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:31.839773 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:31.852957 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:31.893980 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:31.897835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:31.901638 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.907968 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.909766 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.913573 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:31.916545 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:31.921322 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:31.923103 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:31.926949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:31.931852 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:31.943159 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:31.975270 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:31.979832 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.005676 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.015228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.026932 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.029134 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.037492 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.042640 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.085532 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:32.097203 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:32.101008 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.101677 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.107362 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.111918 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.126439 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.127864 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.139221 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.152351 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.152536 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.153679 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.163175 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:32.183117 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:32.206662 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:32.225603 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:32.230957 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.231782 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.233414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.237783 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.260979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.266391 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.320318 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.329195 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.332748 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:32.337446 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:32.338014 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.359521 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.366010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:32.373715 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:32.407126 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.416157 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.418102 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.418494 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.424510 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.429440 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:32.434389 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:32.454241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.454742 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.460906 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.471537 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.473065 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.489053 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:32.502305 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 14:04:32.521935 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:32.529098 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.534175 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.538949 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.554188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.558468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.562482 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:32.594436 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:32.627904 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.642074 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.648428 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.650882 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.653438 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.682230 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.687100 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.700017 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.701572 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:32.704660 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:32.712512 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.712838 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.722894 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:32.735030 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:32.788266 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.788990 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.801477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.815708 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.816506 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:32.823688 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.857335 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.871913 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.874939 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.874955 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.875344 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.877595 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:32.882824 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:32.914453 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.922866 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:32.928730 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:32.931612 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:32.932423 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:32.944705 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:32.948055 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:32.961457 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:32.971767 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 94 similar messages]
W20240502 14:04:32.972465 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:32.989311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:32.989884 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:32.991407 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:32.995319 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.010673 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.024958 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.127364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.139354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.149600 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.153352 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.154582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:33.161863 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:33.168697 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.173322 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.193527 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.200029 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.215402 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.217343 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:33.220484 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:04:33.221063 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 14:04:33.224149 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.237468 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.237913 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.239217 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.242627 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.244899 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.251500 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.271749 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.275568 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.280732 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.290957 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.315110 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:33.326071 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:33.328325 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:33.391260 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:33.392216 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.399317 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.400223 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:33.400466 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:33.424762 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.428623 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.432127 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.476137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:33.479208 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:33.483306 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.495406 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.500810 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.504176 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.506224 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.509589 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.511543 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:33.520232 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:33.520745 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.523022 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.523114 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.578725 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.586846 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.586870 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.594090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.594741 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.595961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:33.599211 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:33.608932 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:33.619760 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.663738 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.679675 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.680011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.685976 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.693414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.702682 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:33.735560 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:04:33.746917 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.750569 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.750735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.751701 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.753515 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:33.772374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.848044 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:33.851577 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.854624 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.865546 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:33.877916 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.881419 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:33.909725 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:33.913180 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:33.928257 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:33.934937 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:33.961253 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:33.962006 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:33.977316 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:33.990872 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.008942 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.010514 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:34.034554 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.054405 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.054559 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.063673 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.076996 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.093062 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.095321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.105868 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.109215 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.113246 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.113508 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.130987 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:34.138411 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:04:34.139169 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:34.139550 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.156330 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.163165 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:34.165460 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:34.191421 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.202553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.202643 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:34.208024 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:34.217993 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.219570 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.220795 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.232986 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.242585 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.246935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.249346 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:34.296957 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:34.310889 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.352452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.355041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.362890 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.377931 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.389475 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.391340 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.400477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.415179 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.415966 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.422621 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.425865 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.427409 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.429373 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:34.433056 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:34.442050 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.447157 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:34.454680 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:04:34.456027 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.458223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.491547 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:34.504703 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:34.542822 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:34.552939 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:04:34.553901 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:34.554837 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:34.556077 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.561331 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.572892 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.572975 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:34.649524 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:34.657039 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:34.660046 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.673971 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.673997 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.674412 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.679064 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.682965 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.687665 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.694763 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.700304 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.702867 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.715690 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.718627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:34.721992 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:34.737736 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.738765 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:34.742473 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.748857 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.754640 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.768252 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.769697 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:34.771066 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.835659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:34.846983 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:34.847262 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:34.855067 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:34.855171 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:34.855753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:34.858912 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:34.880669 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:34.909791 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:34.986311 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:34.992560 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:35.006529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.007826 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.020054 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.021571 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.030040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.031945 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.046226 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.061816 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.063670 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.082203 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.084298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.087157 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.090152 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.091428 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:35.092655 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:35.095101 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.098544 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.100755 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.112432 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.113763 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.138949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.141243 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.141516 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.164368 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:35.209400 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:35.248090 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:35.265825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.275421 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.285985 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.290712 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.295089 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.308466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.309499 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.310781 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.331511 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.333256 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.333925 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.352829 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.354969 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.358726 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.372426 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.376726 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.387514 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.392872 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:35.436933 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:35.501008 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.507422 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.508580 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.522787 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.534180 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.537525 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.542914 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.546218 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.556771 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.562731 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.563287 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.570379 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.580808 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.585424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:35.593947 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:35.597607 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.604655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.605481 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.618844 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:35.687922 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:35.692238 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:35.731840 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:35.744020 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.751309 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.759295 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.760969 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:35.765753 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 14:04:35.770148 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.778577 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:35.781790 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.787268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.798408 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.808168 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.816339 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.822365 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.828810 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:35.840996 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:35.847934 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:35.850142 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.850142 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.850395 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:35.852085 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.865592 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:35.868902 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.871500 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.885252 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:35.894290 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:35.895401 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:35.980937 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:36.012881 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.053084 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:36.070816 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:36.074613 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.075114 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.082671 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.087617 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.098461 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.098594 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.098686 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.098688 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.099967 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.118453 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.123065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.123327 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:36.133249 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 14:04:36.135502 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.150813 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.151528 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.156895 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.158133 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.167966 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.173453 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:36.178788 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.193266 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.194449 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.197960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.203150 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:36.280797 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:36.290853 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.342386 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.371199 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.371742 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.376420 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.386040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.396644 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.401547 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.403529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.404058 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:36.405858 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.412421 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.412619 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.420148 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:36.424753 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.432675 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.435176 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.443176 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.451043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.459568 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:36.464941 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:36.470945 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.471012 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.487624 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.491338 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.498961 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.511510 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.511844 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.520872 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:36.576588 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:36.584349 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.635246 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.637820 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.645740 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.652813 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.659687 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.677325 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.683395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.693282 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.695228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.705444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.706454 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.711777 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.714103 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.718773 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.720448 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.732524 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.734483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.743937 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:36.888831 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:36.900633 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.912171 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.915212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.918143 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.922873 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.934537 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.940405 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.944568 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.944736 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:36.945853 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.956215 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:36.958201 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:36.959996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:36.960206 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:36.963490 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:36.963881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:36.974435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:36.990804 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:37.030628 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:37.033318 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:37.038552 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:37.060745 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:37.061282 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:37.082217 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.083583 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.090281 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.098577 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:37.110682 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:37.118691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:37.136211 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:37.136879 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.141767 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.155220 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.159901 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.160821 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.170881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.177415 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.180462 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:37.215714 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.240948 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.241492 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.243440 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.254529 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.256789 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.257128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.306607 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.312692 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.316273 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:37.323088 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:37.325052 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.337760 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.352604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:37.378700 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.405191 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.406582 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:37.413614 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:37.416971 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.417799 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:37.434166 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.437947 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.438793 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.460435 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.469844 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.471544 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.477890 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:37.483330 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.487653 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.489215 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.561416 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.562839 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.567654 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.568053 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.581044 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.588142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.594347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.598621 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.599483 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.605029 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.607892 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:37.608855 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:37.655938 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.666548 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.683858 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.684398 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.696014 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.698191 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.701107 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.816409 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.823194 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.826027 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.826758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.830986 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:37.833063 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.842021 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.854738 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:37.859798 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.860350 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.861027 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:37.868651 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:37.872814 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.903057 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:37.913156 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:37.922935 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:37.941255 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:37.943785 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:37.949843 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:37.951385 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:37.960638 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:38.004444 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.009877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.014362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.026832 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.030028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.060464 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:38.068681 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 86 similar messages]
I20240502 14:04:38.112632 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.118952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.138131 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.142197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.145239 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.145623 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:38.154163 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:38.155097 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:38.173878 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.187799 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.188436 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.188539 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.196177 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.212144 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.218384 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.223582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:38.228713 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:04:38.245888 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.249585 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.258327 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.275969 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.277499 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.287752 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.304482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.306914 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.319265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.327180 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:38.344240 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 14:04:38.357228 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:38.388865 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:38.395705 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:38.422443 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.427177 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.428609 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.431949 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.443703 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.455466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.455627 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.456912 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.466594 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.472340 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.475668 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.479585 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.547646 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.561771 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.567163 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.574927 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:38.577389 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:38.578719 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.586015 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.608789 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.629318 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:38.642060 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.653306 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.657765 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.663225 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.664690 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.676653 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:38.679523 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:38.687561 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.700767 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:38.703795 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.716980 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:38.755582 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:38.829005 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:04:38.845649 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.850293 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:38.858445 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:38.863512 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:38.865175 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:38.866593 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.867190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.894239 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:38.900336 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:38.912163 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 14:04:38.914485 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:38.928689 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.931982 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:38.939473 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:38.945207 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:38.951061 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:38.956131 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:38.969377 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:38.994812 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.000349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.004686 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.016826 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.018921 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.021767 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.023325 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:39.026146 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:39.036911 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.042665 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.043598 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.052203 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.055816 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.058135 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.059366 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.066126 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.082018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.086673 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:39.103065 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.125842 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.170598 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.171149 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.172953 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.182045 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.184453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.197162 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:39.223871 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.255959 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.257551 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.264343 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.267891 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.271965 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:39.276968 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 14:04:39.277602 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.288071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.311355 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.326551 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.327391 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.335269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.338505 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.351716 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.433300 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.436776 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.441318 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.444180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.444482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.454349 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.456681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:39.478008 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:39.479614 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.480547 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.482415 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.483203 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.490726 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:39.510432 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:39.556202 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:04:39.557214 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:39.564246 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:04:39.571972 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:39.590384 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.595201 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:39.598799 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.602629 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.605604 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.616994 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:39.620532 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:39.655691 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.667640 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.668520 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.669049 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:39.674543 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.682061 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.707243 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.725159 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.728013 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:39.736146 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:04:39.740103 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.743290 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.749553 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.751236 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.755978 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.760951 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.763166 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.770992 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.783965 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.802984 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.807650 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.815248 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:39.885118 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:39.886974 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.889406 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.891566 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:39.894239 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:39.905248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:39.907495 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:39.930801 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:04:39.968348 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:39.973176 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:39.985927 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:39.988941 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:39.994623 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:39.995227 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.014663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.014711 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.027474 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.048349 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.051673 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.063277 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:40.063917 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:40.067636 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.114988 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.126003 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.129369 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.135048 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.146255 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.146751 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.148905 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.156060 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.168747 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.172935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.189327 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.190853 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:40.218518 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:40.266244 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.272205 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.272718 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.280491 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:40.282222 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:40.306885 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.309044 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.314211 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.316179 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.319805 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.348421 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:40.356446 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:40.378142 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:40.385674 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:40.389487 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.393579 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.415549 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.415782 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.422703 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.436305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.438138 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:40.457633 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:40.461266 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:40.488519 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.501863 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.501994 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.506461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.509702 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.510002 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.580333 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.601485 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.606134 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.615556 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.620843 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.627569 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.634467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.645032 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.647451 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.666251 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.683326 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.690316 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.694960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.700879 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.709095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.720736 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.722913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.724910 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.776996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.777240 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:40.799082 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:40.800309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.806421 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.829927 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.830919 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.832819 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.836599 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:40.839663 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:40.850805 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:40.865237 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.865464 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:40.871870 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:40.915936 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:40.940238 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:40.946966 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:40.948478 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:40.959381 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:40.965626 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:40.970613 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:40.980787 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:41.020190 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.030401 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.033020 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.040320 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.040388 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.041553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.045112 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.125829 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.129084 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.136615 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.140872 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.146651 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.164537 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:41.179669 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:41.185436 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 14:04:41.201165 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.201978 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.203871 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.204809 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.213660 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:41.224617 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 14:04:41.231634 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:41.233922 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.242522 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.248873 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:41.257074 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.257758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:41.264364 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.265969 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.268508 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.289265 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:41.307021 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:04:41.347218 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:41.351254 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.401515 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.403712 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.414016 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.414865 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.415555 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.421533 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.423959 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.431891 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.456005 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.457398 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.465344 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:41.466007 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.471041 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.480424 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:41.486340 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.496027 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.498309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.501968 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.505038 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.514478 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.520781 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.534015 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.540478 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.540750 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.546842 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.569473 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:41.584853 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.630394 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.631462 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.639848 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.643268 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.644783 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.646814 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:41.720750 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:41.721616 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.735075 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.737813 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.739939 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.741432 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.742384 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.753237 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.758031 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.758829 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.765632 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.767432 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.794651 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:41.810614 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:41.860100 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:41.867408 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:41.870185 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:41.871608 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.874590 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.884444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:41.891094 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:41.923853 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:41.957648 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:41.977344 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:41.978864 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.982583 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:41.987998 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:41.990319 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:41.991603 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:41.996495 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.998041 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:41.998919 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.000885 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:42.021099 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.037336 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.043732 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.045516 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.046939 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.074256 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.076951 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.083640 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.087649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.091995 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:42.140340 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.205838 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:42.217793 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.217880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.220162 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.245505 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.246827 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.257570 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.257668 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.266691 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.270637 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:42.271006 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:42.275521 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.290328 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.295099 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.303978 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.317066 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.318122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.318591 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.332873 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.343731 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.371709 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:42.381883 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.387621 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.398351 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:42.403371 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:42.403455 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.406545 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:42.410828 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:42.418540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:42.463124 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.479022 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.482760 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.486059 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.486469 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.494796 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.519742 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.563371 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.570797 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.579185 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.583567 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.584933 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.591576 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.633519 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.634797 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.637212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.637800 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.639467 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.656913 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.725193 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.736572 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.738026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.738827 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.739594 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.741503 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.744642 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.759563 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.761771 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.762223 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.787698 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.803167 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:42.824831 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:42.858167 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.865712 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:42.866008 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:42.874570 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:42.907245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.909411 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.955915 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:42.956100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:42.967562 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:42.971376 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:42.972734 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:42.973366 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:42.977057 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:42.979470 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:42.996030 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:43.015353 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:43.052907 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.055217 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.059765 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.066128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.084506 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.090471 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:43.111528 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:43.131917 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.137305 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.142117 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.154881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.161847 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.165452 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.168686 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.170449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.172901 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.193606 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.197590 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.198020 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.198078 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:43.202603 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:04:43.208196 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.212548 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.213135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.215090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:43.222954 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:43.237565 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:43.249274 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:04:43.355892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.360051 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.361294 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.364676 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.370268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.398581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.407539 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.415096 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.426323 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.430553 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.435561 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.452421 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:43.454979 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:43.456486 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.476864 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.479692 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.489037 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.490820 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.494228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.514108 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.514166 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.520623 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:43.532706 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:43.545349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.548736 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.551440 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:43.553798 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:43.564393 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:43.576999 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:04:43.623003 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:43.625283 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.637274 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.641942 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.643611 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.664710 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.665189 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.666649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.667892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.670909 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.676440 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.702590 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.710970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:43.735505 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 14:04:43.782514 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.784333 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:43.786619 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.788439 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.795897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.802179 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:43.826192 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:43.830943 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 14:04:43.852715 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:43.855989 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:43.867194 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:43.891340 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.891660 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.897872 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.906783 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.909226 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.911397 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:43.917903 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:43.935385 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:43.953027 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:43.966584 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:43.969354 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:43.970201 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:43.982256 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:43.982766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:43.995075 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:44.004103 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:44.004925 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.013370 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.017874 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.018429 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.039549 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.088838 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.096371 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.098256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.098441 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:44.109315 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:44.124835 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.126121 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:44.129118 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.130650 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.187301 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.189714 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.196555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.197690 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.199849 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.211417 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.212989 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.220561 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.222213 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.226995 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.246353 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.246891 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:44.301714 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:44.319386 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.323137 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.324817 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.333220 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.342276 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.343401 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:44.358628 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:44.366256 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:44.393651 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.396128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.396885 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.409405 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.410182 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.424785 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:44.445250 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:04:44.447594 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:44.474035 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.474596 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.478473 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.478955 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:44.489615 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:44.492986 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.500892 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.501665 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.503520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.515872 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.516103 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.528264 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:44.536113 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:44.536720 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:44.564915 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:04:44.598008 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.600868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.610532 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.630550 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.631971 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.657196 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:44.663453 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.667346 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.671622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.672830 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.679932 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.686034 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.709506 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:44.729660 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.730646 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.741057 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.742452 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.750607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.755168 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.755563 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:44.843377 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:44.850126 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.857458 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.866956 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.867256 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.872689 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.887317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:44.907723 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:44.913338 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.916446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.924010 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.926908 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:44.928422 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.933873 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:44.940990 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:44.941182 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.956975 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.966377 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:44.977705 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:44.984148 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:44.994527 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:45.034813 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:45.037882 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:45.052227 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.054538 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.057256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.065186 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.075404 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.083056 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.140688 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.143043 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.144773 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.148877 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.167846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.170722 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:45.181885 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:04:45.215124 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.215448 17513 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:04:45.215554 17513 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:04:45.216094 17513 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:04:45.217239 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:45.217335 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:45.217370 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:04:45.217489 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:04:45.217628 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:04:45.218305 17513 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:04:45.225450 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.235435 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.240224 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.244841 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:45.268329 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:45.269907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.281139 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.286932 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.294333 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.298190 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.308528 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.321429 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.352761 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.353682 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.354187 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.368453 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.374686 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.381855 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.431434 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.446688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.449993 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.450417 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.450464 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.450846 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:45.469574 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:45.523638 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:45.524436 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.525665 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.528347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.534339 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.551693 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.555022 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:45.559604 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:45.566643 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.571380 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.584513 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.585208 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.591306 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.591364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:45.595227 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:45.642527 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:45.680943 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.691581 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.708971 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.715126 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.715734 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.716828 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.729228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:45.740787 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:45.747934 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.762907 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.768419 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.780668 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.807741 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.820899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.823434 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.826149 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:45.829362 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.830348 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.833364 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:45.833755 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.835685 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:45.840538 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:45.843487 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:45.861578 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:45.870554 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:45.893985 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:45.899430 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:46.016110 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.019229 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.019706 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.024142 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.032027 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.044231 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.050690 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.054495 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.070127 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.073777 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.077708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.082533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:46.093902 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 14:04:46.098011 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:46.105124 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:46.115314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.115500 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.115782 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.122242 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.126297 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:46.153780 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 14:04:46.154311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:46.171504 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:46.172883 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.189553 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.195159 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.203044 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.212204 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.219226 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.226251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:46.251753 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:46.255985 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.266044 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:46.269446 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.271752 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.281319 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.282634 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.284000 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.296764 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:46.328310 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.339046 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.339797 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.345002 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.347309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.350270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.366407 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.440789 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.444768 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:46.453502 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.457964 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.464691 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.480566 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.492992 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.525352 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.536101 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.541831 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.544816 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.549546 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.559013 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.576030 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.583319 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.590667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.591643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.602409 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.605053 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.617461 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:46.624100 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:46.641942 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.650105 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:46.655042 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:46.655542 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.666141 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.673285 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:46.711515 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.786906 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.791527 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.793855 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.802572 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.810997 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:46.817919 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.821276 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.848022 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:46.849474 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.851902 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.854936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.865751 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.874424 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.875298 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:46.883793 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.885303 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.889081 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.891795 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:46.894013 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:46.898113 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:46.905702 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:46.932314 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:46.986235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:46.989781 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:46.990864 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:46.990927 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:46.992214 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.025545 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.053368 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.090826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.094724 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.095449 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.097808 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.115047 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.118213 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:47.128790 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:47.136914 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.139950 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.140868 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.148706 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.156038 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.166107 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.187011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.194821 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.196329 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.203503 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:47.218688 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:47.218823 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:47.227342 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.304723 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.308424 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.312712 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.329522 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.332794 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.332957 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.343086 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.350443 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.351362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:47.355913 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:47.357021 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.373847 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:47.374110 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:47.374581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:47.399922 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:47.403650 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:47.417061 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.423319 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.426285 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.428886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.429726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.437215 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.517963 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.518515 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.524981 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.526069 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.534149 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.553108 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.592265 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:47.594065 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:47.594971 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:47.596052 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.603617 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.615535 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.616395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.621615 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.626365 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.640132 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.648954 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.671749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.675438 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.697100 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.701968 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.702673 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.725868 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.726075 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.729568 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:47.736480 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:47.750897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:47.772805 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:47.815685 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.816903 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.820325 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.835227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.836478 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.856341 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:47.871867 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:47.915096 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.923897 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.923934 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.925994 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:47.931130 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:47.936236 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.936918 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.952571 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:47.960775 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:47.963135 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:47.987828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:47.996930 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.032197 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.036761 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.042610 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.047341 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.051831 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.074023 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:48.076987 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.123268 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.132787 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.133713 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.143796 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.145365 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.146961 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.147475 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.153357 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.162039 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.172255 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:48.172438 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:48.184847 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.186476 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:48.192554 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:48.216542 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240502 14:04:48.234692 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:48.251540 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:04:48.278203 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:48.280635 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:48.290264 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.296705 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.301146 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.323490 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.323868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.327067 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.336146 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.338968 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.338994 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.353490 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.371295 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.433010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.439934 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.443490 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.444159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:48.446755 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:48.454005 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.457198 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:48.497035 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.538185 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.538735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:48.545992 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.552897 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.557476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.562203 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.563042 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.569473 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:48.595657 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:48.603374 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.604203 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.606313 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.609858 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.634425 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.636083 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.652592 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.653034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.656960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:48.663681 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.665799 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.679436 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.695783 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:48.696427 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 14:04:48.712469 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.713213 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.717018 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.718181 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:48.719921 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:48.731758 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.732381 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:48.769933 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 14:04:48.833853 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.845757 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.860536 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.861725 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:48.868714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:48.871560 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:04:48.876112 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.890900 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.923375 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.929558 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:48.935148 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.945029 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.952340 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:48.957609 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.958433 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:48.960857 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:48.963969 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:48.968818 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:48.975878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:48.983057 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:48.986898 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:48.994479 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:48.997810 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:49.005278 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.023406 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.035431 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.035672 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.049482 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.066828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.067195 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:49.087713 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:49.117350 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:49.141789 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.142529 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.149051 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.150467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.150758 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.165421 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.220162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.224568 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.232745 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.252830 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.256386 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.257087 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.258472 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.267441 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.268599 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.269230 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.269281 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.289772 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:49.313786 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:49.368300 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.380287 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.384321 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.391320 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.397960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.401021 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:49.428839 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:49.461342 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.465540 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.467550 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:49.475544 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:04:49.476167 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:49.480815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.487519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.502095 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.506577 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.513804 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.514416 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.514555 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.519929 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:49.521011 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:49.538581 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:49.558678 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:49.586200 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.591135 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.601105 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.605917 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.609845 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.613085 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:49.656827 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:04:49.658897 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:49.675894 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:49.699667 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.708473 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.711755 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.714228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.723348 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.726313 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.737605 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.743876 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.756521 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.768728 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.782442 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.787362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.826514 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.839040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.841420 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.865937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:49.868855 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.871860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:49.891170 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:49.900980 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:49.933492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:49.937812 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:49.949461 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:49.950707 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:49.954912 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:49.960455 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:49.972522 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:49.976202 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:49.984365 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:49.992007 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.000910 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.005321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.005575 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.027063 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:50.047194 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:50.102314 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.123577 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.130244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.146853 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.147008 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.148547 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.159735 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.163889 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.168743 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:50.169723 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:50.178439 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.178973 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.201782 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.232360 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.235123 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.237962 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.239434 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.244587 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.251838 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.303567 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.320055 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.322804 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:50.326856 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:50.328375 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.330976 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.333205 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.357400 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.359736 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.363842 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.368484 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.372082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:50.373469 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:50.383193 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:50.406900 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:50.410341 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:50.434746 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:04:50.485000 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.501152 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.510717 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.517472 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.523793 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.524085 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.533844 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.536494 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.536733 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.545964 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.556594 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.558898 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.567646 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.568707 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.571339 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.572719 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.592190 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:50.594664 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:50.601291 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.675917 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.675980 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.684804 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.691591 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:50.709394 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:50.713083 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.726851 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.730543 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.736878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:50.739360 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:50.741762 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 14:04:50.744720 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:50.756637 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.760368 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.762423 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.769351 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:50.803052 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:50.816171 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.821484 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.828590 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.844475 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.847128 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.848496 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:50.932683 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:50.945758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:50.945925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:50.966559 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:50.979750 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:50.980551 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:51.001124 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:51.006172 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:51.031090 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:51.042192 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.058668 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.070756 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.077476 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.081470 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.088760 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.090278 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.101579 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.103960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.116034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.139389 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.148541 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.151335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.156961 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.159021 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.163725 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.173173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.173383 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.176358 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.193598 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.199075 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:51.203410 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:51.204532 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.214943 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:51.229017 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.233983 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:51.296941 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:51.311352 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.316592 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:51.319744 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:51.323148 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.330276 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.332408 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.344401 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:51.369246 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:51.385357 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 14:04:51.399438 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.405344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.414732 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.416172 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.416631 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.417181 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.421336 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.426010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.427876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.428158 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.432559 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.436427 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:51.446731 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.449627 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.485868 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.546483 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:51.592408 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.607188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.611229 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.616914 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.618760 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.627758 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.652016 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.655941 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:51.661572 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.662220 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:51.664474 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.665258 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.673734 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.674986 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.679404 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.686183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.695467 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.713618 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.718878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.726526 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.729010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.742895 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.752467 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.759425 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.765107 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.765372 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:51.775313 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:51.812978 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:51.895077 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:51.908046 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.913996 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.924726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.935004 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:51.943442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:51.944548 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:51.949504 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:51.950448 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:51.974800 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.986151 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:51.986773 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.011821 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.036721 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:52.048547 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:52.052409 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.056046 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.063040 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.069134 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.069743 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.071291 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.071717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.078076 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.089748 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.091758 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.102905 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.177506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:52.190189 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:52.204738 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.208544 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.209503 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:52.245024 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:52.245308 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.251154 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.256210 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:52.276757 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:52.285847 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.303404 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.303706 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.306087 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.314064 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:52.318295 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:52.363162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:52.374404 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:52.374639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.374851 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.377758 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.378657 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.379415 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.409369 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.413940 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.426282 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.442425 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.447160 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.447494 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.480513 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.482087 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.490177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.491263 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.494441 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:52.505088 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:52.506011 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:52.569150 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:52.580838 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.596796 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.601536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.621088 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.627096 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.638233 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.646138 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.648417 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.651782 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.664391 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.667567 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.678735 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.721406 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.727650 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.747982 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:52.761890 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:04:52.762023 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:52.768085 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.768175 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.772883 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:52.779043 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:52.779743 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.788976 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:52.809212 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:52.819716 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.822516 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.823482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.834370 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.888248 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:52.893493 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.918417 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:52.919555 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.920634 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:52.922195 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:52.924132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.931005 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.933043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:52.938030 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:52.941233 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:52.944353 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:52.949426 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.056452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.056556 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.063156 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.068622 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.073586 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.081379 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.135141 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.145720 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.162936 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.166114 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.176688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.185523 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:53.194062 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:53.206276 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:53.209875 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.232102 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.233629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.249264 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.250698 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.251684 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.253108 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.254007 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.256001 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:53.262125 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:04:53.262778 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:53.266934 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:53.275537 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:04:53.276418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:53.280041 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:53.285212 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:53.303009 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:53.304329 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 14:04:53.330117 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:53.353317 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:53.354837 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:53.358548 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.363971 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.366887 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.374137 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.374313 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.449877 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.459697 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.460216 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.462281 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.470583 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.479962 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.524251 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.531319 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:53.533150 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:53.538131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.541152 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.567349 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.570994 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.578317 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.596010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.600788 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.608008 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.612651 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.614781 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:53.628360 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:53.636785 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:53.675189 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.682780 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.684437 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.700341 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:53.700341 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:53.702188 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.709122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.729543 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:53.734668 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:53.735008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.738664 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.741663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:53.747599 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:53.750790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.763955 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.836093 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.840973 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.843240 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.846777 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.848960 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:53.859777 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:53.862793 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:53.864638 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:53.866217 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.867111 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:53.869560 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:53.873183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.876519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.900907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:53.929454 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:04:53.930016 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:53.968868 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 14:04:53.973281 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:53.978343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:53.989571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:53.990794 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:53.994587 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.005172 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:54.054889 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:54.086858 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.093559 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.096614 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.103071 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.114353 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.117576 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.123545 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.124002 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.131726 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.156386 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.167826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:54.178440 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:54.179510 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.189949 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.198891 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.203361 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.219516 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.234403 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.242913 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.254639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.255909 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.259927 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:54.268761 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:54.275106 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.289646 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:54.303961 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:54.308368 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:54.337165 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:54.338173 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.340695 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.340925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.347348 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.353698 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.369342 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.434856 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.435304 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.439879 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.440639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.441473 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.441515 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.514736 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.520097 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.529018 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.530467 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.538544 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.548926 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.579450 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.584373 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:54.594358 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:54.612529 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.617188 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.623090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.628943 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:54.639046 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:54.646111 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.654505 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.663168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.663700 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.666438 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.668642 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:54.670307 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:04:54.672672 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:54.688499 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:54.747918 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:54.753703 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:54.755765 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.765791 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.770470 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.775707 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.783591 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:54.793820 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:54.824373 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.827491 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:54.836978 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:04:54.838023 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.838654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.838668 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.839226 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.901921 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.914932 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.917090 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.918792 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:54.923368 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.927039 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.950839 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:54.951228 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:54.957321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:54.959056 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:54.977514 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:54.998769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:55.049489 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:55.082278 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.086133 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.090734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.091688 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:55.109062 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:55.116328 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.125677 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.147512 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.149912 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.157716 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.164341 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.176612 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.177935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:55.219998 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:55.223832 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:55.227969 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.248636 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.255455 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.256752 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.268594 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.274271 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:55.284782 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:55.290309 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.298223 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.300083 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.308897 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.314868 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.321442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.332926 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.356079 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.356559 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.359632 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:55.366839 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:55.368942 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:55.385530 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:55.398370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.435890 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.438997 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.441864 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.443068 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.458135 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.460335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.509891 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.521297 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.522162 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.523432 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:55.532974 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:55.542157 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.556043 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:55.604832 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:55.627394 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.629736 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.631796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.638116 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.646396 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.658023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.671329 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.671676 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.673861 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.682423 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.691977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:55.699685 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:55.715646 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.722973 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:55.725611 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:04:55.730509 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.730628 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.733659 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.738881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.756121 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:55.761658 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:55.782974 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:55.815367 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:55.857489 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:55.873636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.880882 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.889535 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.892460 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.898550 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.899896 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.911072 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:55.911283 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:55.917838 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:55.921891 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.923230 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.925765 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.929580 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:55.935906 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:55.943635 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:55.960441 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:55.988392 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:55.989171 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.002593 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:56.010185 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:56.012585 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:56.036746 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 14:04:56.040777 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:56.046767 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.050719 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.051383 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.056910 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.058429 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.082551 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:56.108474 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:56.125124 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.125690 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.130630 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.133139 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.147119 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.148999 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.192786 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.196413 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.198127 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:56.200853 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.209362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.210634 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.236243 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.277714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.278077 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.284035 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.296460 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.301546 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.305814 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:56.359886 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:56.394691 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.395156 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.399278 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.401151 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.404664 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:56.405452 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.411147 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.444190 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.448236 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.454705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.470206 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.472353 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.504349 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:56.507565 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:56.511257 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:56.525198 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.525903 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.527894 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.542140 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.544950 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.556200 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:56.560911 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:56.603564 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 14:04:56.635048 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:56.637512 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.641731 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.656220 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:56.671851 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.673774 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.674389 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.684405 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.688760 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.692911 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.697392 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.702479 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.702623 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.708874 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.714562 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.714902 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.746610 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.753294 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.760989 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.765233 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.844688 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.848169 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:56.860164 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.863180 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.872941 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:56.879586 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.884306 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.885927 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:56.891111 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:56.905501 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:56.932682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:56.946799 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:56.956881 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:56.971906 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:56.999680 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.014616 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:57.015426 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:57.016559 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.023658 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:57.026719 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:57.027463 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.045476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.051995 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.054327 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:57.057061 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:57.057729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.077905 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.091393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.098451 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:57.182011 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:57.189838 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.195039 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.196434 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.197681 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.203088 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.205024 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.205312 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.206071 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.213788 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.219741 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.225440 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.232388 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.235322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.251758 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.271013 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.288705 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.289397 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.296438 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.363062 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:57.363432 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:57.366475 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:57.373467 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:57.373951 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.378439 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.379928 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.381511 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.424193 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.432355 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:57.438560 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:57.449501 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.449724 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.451009 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.457929 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:57.480343 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:57.506155 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.511210 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.515965 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.517149 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.518347 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.529465 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.565084 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.583060 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.589686 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.589785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.592365 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.605187 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:57.636930 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:57.663447 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.663717 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.672923 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.682969 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.685994 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.694937 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.699349 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:57.718164 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:57.724766 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.728222 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.733070 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.733094 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.745338 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:57.833616 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.835013 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.853818 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:57.854120 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:57.857393 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:57.863886 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.882071 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:57.886104 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:04:57.892169 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.895850 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:57.896960 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:57.898809 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.912299 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:57.913857 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:57.935592 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:57.982721 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:57.989816 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:57.994138 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:57.998008 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.002756 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.005647 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.033605 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.033895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.045193 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.045598 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.046779 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.062489 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.074402 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.085085 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:58.129810 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.135565 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.142617 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.146344 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.152364 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.168331 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.171655 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.174628 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.182657 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:58.184876 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.185770 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.187831 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.190277 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.217657 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:58.217777 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:58.219216 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:04:58.244939 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.275959 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.281607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.282761 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.284458 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.299233 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:58.313818 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:04:58.321295 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:58.324913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.386328 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.387324 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.392771 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.403641 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.408180 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.409150 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 93 similar messages]
I20240502 14:04:58.423103 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.423501 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.428865 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.433395 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.436975 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.445925 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:58.447422 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.454201 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.455874 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:58.464419 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:58.466318 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.480713 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.488613 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.493872 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.510895 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:58.533951 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:58.562649 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.567790 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.567786 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.569792 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.577229 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.582779 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.628783 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:58.682096 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:04:58.694247 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 14:04:58.698350 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.698576 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.699568 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.700682 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.703680 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.727936 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.731192 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.736419 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.745064 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.755246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.757717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:58.758479 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 14:04:58.772014 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.780426 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.788364 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.794211 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:58.798173 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.808224 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.818312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.825479 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:58.844194 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.883929 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:58.885345 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:58.893566 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.896708 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.898069 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:04:58.919963 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:58.920430 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:58.920502 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:58.972457 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:58.978605 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:58.979077 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:58.979182 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.000234 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.011328 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:59.049834 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:04:59.082031 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.086377 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.099848 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.102182 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.105865 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.108145 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:59.145277 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:59.148615 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.149717 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.150504 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.150653 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:59.157312 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:59.157925 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:59.165375 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:04:59.179880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.207312 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.217779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.218474 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.221128 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:59.225980 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:59.246421 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.260043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:59.260398 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:04:59.348533 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.363880 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.367815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.380666 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:04:59.381490 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:04:59.384141 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.384477 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.385015 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.386027 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.389719 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.390113 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.399329 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.403714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.408439 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.419750 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.424937 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.432132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.448782 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.451962 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.511664 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.514565 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.516943 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:59.521461 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:04:59.526191 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:04:59.526248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.531778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.555968 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.617183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.634960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.640448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.643421 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.654232 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:59.656673 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:04:59.657845 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.682226 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.683737 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.683841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.694614 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.705015 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:04:59.715750 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:04:59.718183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.770988 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.772874 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.775663 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.778529 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.779182 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:04:59.779765 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:04:59.818094 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.820667 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:04:59.825522 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:04:59.827682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.860970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:04:59.869185 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.870914 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:04:59.871156 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.872642 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:04:59.873216 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.886960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:04:59.889753 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:04:59.901417 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:04:59.913277 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:04:59.913398 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:05:00.003543 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.011375 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.027616 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.035794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.038812 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.074077 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:00.077822 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.081454 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.085750 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.087313 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.111872 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.116218 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.119436 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:00.123387 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:00.145804 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:00.155849 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.157418 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.158923 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.162015 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.164209 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.166277 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.174235 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.192004 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.206576 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.209851 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.221275 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.233223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:00.250695 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:00.323534 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:00.343497 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:00.344020 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.344494 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.344761 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.345106 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:00.346386 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.347966 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.354771 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.366997 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.370308 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.379091 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.380635 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.422570 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:00.431731 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.439473 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.443531 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.456475 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.458170 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.469592 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.481547 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:00.517031 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:00.526888 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:00.527709 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:05:00.563961 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.564486 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.568115 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.572219 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.585085 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:00.585551 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:00.599032 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.622748 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.642167 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.646710 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.652887 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.658591 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.661787 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:00.669713 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.673238 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.686830 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.695647 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.696511 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.699334 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.700291 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.704918 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.709450 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.724412 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:00.724730 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:00.727728 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.742520 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.754428 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:00.758013 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:00.771489 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.866418 25544 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:00.866583 25544 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:00.867214 25544 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:00.868257 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:05:00.868428 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:05:00.869418 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:00.869518 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:00.869553 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:00.869937 25544 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:05:00.871001 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:00.903668 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:00.907490 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.911715 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.912389 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.924337 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.932591 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.934378 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:00.950335 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:00.957784 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:00.962643 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.963636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:00.964638 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:00.969525 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:00.977070 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:00.977399 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:00.998548 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.000232 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.025095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.047719 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:01.071264 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:05:01.071480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:01.071920 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.082341 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.084810 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.090301 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.111177 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.118243 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:01.162838 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 14:05:01.171747 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:01.182631 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:01.190404 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 14:05:01.196399 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:01.199632 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.202390 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.207151 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.209728 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.215920 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.219494 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.222580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.225397 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.230314 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.266443 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.268922 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.289662 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.303222 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.304064 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.306052 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.316418 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:01.333765 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.337178 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.364044 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:01.368777 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:01.369334 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:01.420146 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.422197 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.422969 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.432502 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.437707 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.461162 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:01.519351 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:01.536180 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.543911 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.544364 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.549517 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.550843 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.569084 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.569499 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.570858 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:01.584975 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.587750 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.595270 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.602118 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:01.618256 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:01.623577 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.630893 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.632978 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.640678 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.643752 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.644784 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.658938 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.669977 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.737227 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:01.747366 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:01.747822 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.756534 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.758209 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.759179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.767637 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.776366 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:01.782027 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:01.790377 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:01.793421 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.796252 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.798244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:01.826540 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:01.831414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.859825 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:01.870734 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.876132 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.877868 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:01.879030 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:01.907802 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:01.909673 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:01.992770 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:01.993836 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:01.996534 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.003549 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.005024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.005272 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.010533 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.011829 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.035238 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.047178 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.049705 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.053632 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:02.082983 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:02.127094 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.130061 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.132536 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.145329 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.145447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.155936 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.177739 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.180109 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.181279 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.183775 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.202590 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.213048 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.239028 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.254961 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.257536 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.282575 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.295390 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.318434 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:02.331422 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:02.369935 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:02.381935 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:02.387076 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.387639 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.404960 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.432691 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.444041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.445137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.445785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.452013 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.456590 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.461519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.469211 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:02.483814 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:02.486423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.504997 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.517652 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.523814 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.535794 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.537293 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.559592 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.568802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.568807 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.569403 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:02.576117 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:02.591150 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:02.601225 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.601594 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.614444 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.623682 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.639816 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.651621 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.663849 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.667490 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:02.717712 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:02.739562 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.749929 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.752048 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.754194 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.754323 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:02.773571 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:02.783176 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:02.785593 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:02.799901 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.800541 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:02.810590 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.812038 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:02.824748 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.846343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:02.848197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:02.853307 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:02.853317 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:02.859895 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.862761 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:02.863580 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:02.880921 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:02.889662 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:05:02.964207 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:02.983614 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:02.993341 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.002995 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.015681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.019219 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.020203 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.021531 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.022424 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:03.029958 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:03.035001 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.037277 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.046051 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.050271 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.058758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.070235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.071450 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.072518 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.081830 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:03.087235 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:03.088893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.092248 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.096503 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:03.113809 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:03.184703 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:03.211138 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.212680 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.222062 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.225618 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.239503 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.239522 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:03.247812 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 14:05:03.265610 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.268748 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.316321 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.317404 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.325371 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.329773 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.338766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.340157 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.340899 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.356066 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.357304 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.364953 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.369378 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.385090 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.419296 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.423365 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.423346 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.423568 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:03.426465 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:05:03.448346 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:05:03.460995 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.506673 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.514257 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:03.520344 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:03.523072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.524910 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.526176 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.539369 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.573890 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.578331 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.583571 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.586556 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:03.598590 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:03.598974 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:03.605329 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.606225 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:03.621357 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 14:05:03.627653 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:03.646952 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.664166 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.665861 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.677071 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:03.678696 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.679488 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.679706 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.680377 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.690793 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.702598 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.710131 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.711683 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.729192 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.738420 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:03.804728 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.835278 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:03.837019 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.843137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.872442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.872648 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.873184 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.873885 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:03.891330 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.912804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.917318 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.918802 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:03.931414 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.934448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:03.935175 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:03.941277 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:03.941743 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.945757 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:03.962392 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:03.965740 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:03.970906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:03.973719 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:03.997519 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:04.040146 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:04.093653 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.095795 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.096719 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.099629 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.109488 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:04.115355 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:04.132813 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.135741 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.143007 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.151595 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.154556 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.162142 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.162261 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.166117 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.171535 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.173018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.179457 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.207338 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.209501 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.273303 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.278550 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:04.284628 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:05:04.286629 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.291935 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.293159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:04.293992 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:04.294194 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.333717 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:04.336648 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:04.340180 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.345753 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.356104 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.366976 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:04.371048 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:04.371444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.429603 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.431972 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.432478 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.437337 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.441077 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.441977 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.454000 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.456449 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.460206 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.464689 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.490599 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.508649 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.561060 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.561302 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.565258 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.569260 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.570122 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.580889 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:04.597973 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:04.638048 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.639137 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.644675 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.651342 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.653652 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.658358 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.709542 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.710604 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.713220 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.716387 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.723984 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:04.729876 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:04.738736 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.849103 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.849241 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.851644 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:04.865136 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.871723 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.873634 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.883252 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.891356 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:04.897713 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:04.904359 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:04.905617 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:04.913204 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:04.917443 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.928714 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.934809 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:04.935334 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:04.937228 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:05:04.940961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:04.946468 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:04.955678 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 14:05:04.987439 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.034197 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:05.034701 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.038239 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.038537 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:05.046841 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:05.048774 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.068765 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.072533 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.105844 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:05.133831 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.135182 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.136705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.143590 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.145196 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.171525 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.175444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.178246 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.179750 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.180506 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.180539 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.186744 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.210623 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:05.226219 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:05.239239 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:05.241240 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:05.243664 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.245904 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.250047 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.269199 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.286999 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.289125 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.356915 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.356959 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.359104 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.364163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.365376 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.378352 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:05.397224 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:05.406857 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:05.420004 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.423746 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.424677 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.425072 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.442685 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.445432 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:05.486363 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.494342 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.498615 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.510807 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:05.513300 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.521097 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.533977 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.536106 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:05.567564 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:05:05.620157 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:05.623126 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.635294 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.645344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.647109 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.648531 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.649598 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.671711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:05.685952 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.704306 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.709286 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.711201 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.715003 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.716830 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.719980 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.723939 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.724681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:05.725411 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:05.744468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.752278 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.759553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.772536 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.783432 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.810604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.810643 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:05.836580 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.839879 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:05.867323 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:05.884112 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.887905 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:05.906090 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:05.909015 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:05.911219 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:05.912165 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:05.915768 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:05:05.941931 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:05.950188 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 14:05:05.979672 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:05.980422 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 14:05:05.993397 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.994714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:05.998324 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.009896 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.023558 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.035385 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.041312 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.047389 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.063793 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.082262 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.109270 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.113227 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.120739 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.120815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.121778 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:06.122059 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:06.129978 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.148772 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.158138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.210219 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.212848 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.213806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.215943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.224131 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.234531 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.238286 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.240738 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.244105 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.246253 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:06.250244 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:06.265321 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:06.281368 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:06.282248 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.282286 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:06.292845 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:06.302969 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:06.390115 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.393613 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.398602 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.403674 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.408942 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.410102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:06.420363 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:06.440320 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:06.444860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.451442 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.452105 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.458204 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.472465 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.477349 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:06.517859 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:05:06.558614 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:06.568679 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.569309 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.569968 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.573925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:06.583289 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:06.590490 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.616427 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.624363 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.628041 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.628669 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.633618 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.669533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.672025 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.690825 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.698725 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.705649 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.714313 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.729444 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.733135 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.800105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.804806 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.812343 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.824749 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.843387 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:06.848488 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:06.850729 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.872368 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.882925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.884526 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:06.891395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.894969 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.896953 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.939680 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:06.942664 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:06.947335 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:06.947878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:06.948765 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:06.949380 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.032606 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.040658 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:07.055483 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:07.055871 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.062726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.067698 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.069986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.082849 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.095422 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.102828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.111137 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.117771 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.121163 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.121433 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:07.136040 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.156039 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.156339 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.157461 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.159574 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.181767 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.196589 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.245458 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.247362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.251169 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.262356 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.270288 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.282845 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.312773 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.325642 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.325795 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.333226 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.345836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.354116 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:07.368963 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:07.395227 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.399302 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.404742 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.405206 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:07.407440 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:07.408721 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.410353 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:07.444990 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:07.517572 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.534078 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.538856 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.542444 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.550197 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:07.558655 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.563661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.581254 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.583165 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.587924 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.592133 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.593798 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.603422 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:07.606587 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.607125 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.612514 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.637905 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.650089 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.656905 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.662150 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.665549 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.666523 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.669366 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:07.673218 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:07.683166 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.692981 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.702332 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.703445 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.708150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.711817 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.726506 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:07.736093 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.736995 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.739609 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.740828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:07.788739 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:07.820314 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:07.876691 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.877135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.881358 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:07.888931 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.892393 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.907766 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:07.920532 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:07.929373 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:07.945449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:07.954144 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:07.960022 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:07.974404 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:07.974521 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:07.981209 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:07.982404 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:08.016405 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.016716 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.016716 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.030035 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.040592 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.042826 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.104786 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.105643 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.108428 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.114498 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.117036 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.136533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.157037 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.160821 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.166702 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.175199 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.182545 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.190047 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.247010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.252367 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.254717 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.254933 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.261292 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.261672 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:08.266892 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:08.297701 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:08.315284 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:08.364892 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.371464 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.371574 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:08.374285 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:08.380028 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.382891 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.383610 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.415174 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.419198 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:08.422803 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 14:05:08.431769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.434870 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:08.435395 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 14:05:08.450062 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:05:08.450690 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:08.460793 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:05:08.461498 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.469318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.471241 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.471814 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.474050 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.480813 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.488065 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.496644 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.497599 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.514494 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.527585 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.544495 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.548777 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:08.556599 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:08.565467 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:08.569198 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:08.579313 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:08.590559 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:08.599915 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:08.622172 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:08.625083 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:08.682308 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.691707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.693473 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.712615 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.715615 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:08.728065 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:08.730240 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:08.733300 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.734158 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.734714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.757513 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.757699 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.757722 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.764465 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.768543 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.777024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.792438 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.792754 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.793103 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.807701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.809672 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:08.830353 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:08.831912 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.847020 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.852663 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:08.869194 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.870007 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.929404 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:08.939786 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:08.941350 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:08.950798 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:08.961452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:08.975019 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.059274 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.061627 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.063889 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.069269 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.086387 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:09.096861 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:09.101131 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.101320 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.109078 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.114558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.115990 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:09.118115 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:09.128453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.135152 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.191048 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.192095 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.196293 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.210258 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.212749 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.224743 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.294703 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.297259 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.302376 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.308141 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:09.310017 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:09.332379 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.343492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.364899 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.367844 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.368453 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.372025 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:09.391409 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:09.391522 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:09.405038 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:05:09.408452 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.430573 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.444244 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.447006 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.457453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.464468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.469199 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.471020 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.479341 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.497386 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.503257 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.508632 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:09.510869 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:09.522423 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.584558 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.595582 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.597296 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.603446 31506 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:09.603579 31506 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:09.604116 31506 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:05:09.605934 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:09.606050 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:09.606096 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:09.606629 31506 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:05:09.607003 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:05:09.607131 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:05:09.611758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.626791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.629066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.630829 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:09.642612 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:09.645993 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.646731 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.658944 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.660918 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.670462 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.726195 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.728925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.729045 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.731963 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.742807 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:09.754415 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:09.757122 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.816294 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.820462 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.822161 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.824915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.834640 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.849556 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.903935 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.906911 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.907699 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.908388 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:09.911113 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.917030 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.941080 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:09.944916 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:09.946483 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:09.947562 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:09.949028 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:09.958050 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:09.972626 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:10.007045 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:05:10.024487 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:10.036119 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:10.039472 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.047335 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.054515 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.067391 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.070017 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.081010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.106351 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.106431 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.125074 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.128970 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.133241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.134423 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:10.147153 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:05:10.147827 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:10.158003 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:10.181978 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:10.214560 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.216327 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.223637 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:10.226215 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:10.230496 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.236202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.236547 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.238314 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.248831 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.252885 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:10.284986 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:10.288435 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.291989 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.308533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:10.333783 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:10.362314 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:10.365713 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.374254 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.400336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:10.400964 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:10.401518 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.408176 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.418938 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.424185 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.429659 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.437018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.439281 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.445923 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.468582 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.475739 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.490689 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.514211 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.524336 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.530006 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.532410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:10.536077 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:10.566641 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.573958 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.574524 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.599355 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.601578 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.610152 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:10.662710 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:10.672511 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:10.673933 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:10.678565 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.692241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.698320 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.698989 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.708644 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.708972 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.714754 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.736889 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.737115 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.737339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:10.748163 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:10.754369 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:10.757508 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 14:05:10.765581 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.766137 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.771900 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.777159 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.783411 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.802881 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:10.838886 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:10.875619 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:10.906862 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.907101 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.908593 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:10.920053 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.940019 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:10.947551 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:10.948249 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:10.950153 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:10.958122 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:10.959682 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.960608 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:10.967437 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:10.968851 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:10.993642 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:11.004556 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:11.008621 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 14:05:11.056267 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:11.067711 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.077380 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.079370 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.081481 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.094882 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:11.114102 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:11.116511 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:11.120642 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:11.124872 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.149288 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.152410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.168488 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.168823 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.169456 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.205353 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:11.224411 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:11.228525 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.235126 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.248044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.260850 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:11.261268 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:11.265198 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.274046 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.277230 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.280790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.283629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.287833 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.288568 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.308682 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:11.331213 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:11.332911 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:11.350859 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.359426 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.362262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.374194 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.394496 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.400915 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.426267 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.431775 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.432509 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.440982 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.441028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:11.470911 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:11.476121 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.503248 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.516121 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.518326 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.521385 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.543823 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.548462 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.607172 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.611841 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.615464 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.620715 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.631428 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.648456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:11.684787 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:11.698308 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.701934 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.705397 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:11.707757 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:11.707870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.725322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.746363 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:11.766611 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:11.771903 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.779714 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:11.781330 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:05:11.784615 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.789203 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.790149 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.818748 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.888267 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.909890 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.912114 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.915486 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:11.915992 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:11.932991 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.941670 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:11.944097 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.945657 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:11.946341 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:11.969370 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:11.972211 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.016244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.016470 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.024076 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.024525 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:12.026877 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:12.037132 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.055040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.081073 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.087620 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.090574 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.098567 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.108860 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:12.113742 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:12.117337 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:12.174178 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:12.203766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.213008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.216763 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.228255 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.229638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.229928 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.264660 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.266897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.269767 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.270262 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.276865 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.297880 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.306862 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.310770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.322855 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.337741 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.340821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.349455 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.357782 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.358055 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.359823 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.372977 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.400570 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.406172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.439754 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.440049 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.443670 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.452697 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.453971 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.461519 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:12.486776 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:12.501864 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:12.536554 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.537676 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.539676 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.542522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.543058 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.551085 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:12.565781 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:12.586794 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:12.648524 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.650043 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.655455 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.657670 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.659255 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.672231 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.750581 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:12.762351 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:12.762790 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.769593 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.779174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.780431 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.787055 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:12.788789 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:12.794322 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:12.794741 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:12.796604 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:12.812100 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.813139 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.825698 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.829174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:12.831075 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:12.841446 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:12.843511 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:12.861075 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:12.863782 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.870751 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:12.872809 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:12.895138 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:12.895803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:12.924381 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:12.966444 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:12.971185 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:12.977306 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:12.982332 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:12.994122 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.000450 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.003423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:13.011368 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:13.012451 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.016438 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.016616 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.037477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.039585 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.041762 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.101150 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.106485 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.111122 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.118191 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.137130 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.147028 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:13.152699 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:13.153787 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.160665 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.172106 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.172443 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.173202 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.209815 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.242702 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.248433 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.249467 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.252404 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.253943 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.261698 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:13.293249 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:13.324700 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.325636 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.327389 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.329602 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.331847 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:13.349471 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:13.349787 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:13.357671 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 14:05:13.360026 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:13.393941 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.396421 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.412339 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.413311 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.416079 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:13.416666 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:13.426473 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:13.436841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.458024 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.470036 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.471930 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.472142 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.484030 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.492866 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:13.515115 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 14:05:13.529182 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:05:13.530910 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:13.531762 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:13.551764 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.552843 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.553174 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.565547 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.581476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.602665 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:13.618654 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:05:13.638227 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:13.654820 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:13.678540 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.699664 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.708895 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.709084 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:13.717427 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:13.717746 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.727769 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.732573 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.732746 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.749552 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:13.749929 2952 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:13.750027 2952 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:13.750458 2952 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:13.751762 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:05:13.751881 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:05:13.756856 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:13.761538 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:13.761662 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:13.761698 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:13.762063 2952 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:05:13.763260 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.764940 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.765681 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:13.765992 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:13.766824 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.785154 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.785684 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.787007 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.787731 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.793856 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.808444 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.813045 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.833730 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:13.834348 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:13.836480 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:13.838711 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:13.864924 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:13.865309 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:13.945428 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:13.952809 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:13.954476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:13.960045 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:13.960355 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:13.985345 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.028585 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.031483 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.041288 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.041389 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.047776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.051487 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:14.053083 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:14.055683 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.090633 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.094555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.095450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.097268 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.106186 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:14.164585 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:14.173385 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.174132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.182261 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:14.190276 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:14.190979 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.196143 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.196408 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.252842 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.262835 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.263696 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.266216 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.273154 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.288043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.299381 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.299643 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.300344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.307960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.324611 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.332541 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:14.342629 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:14.367666 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:14.409358 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:05:14.425487 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.436254 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.438624 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.454907 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.455552 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.466529 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.491897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.496521 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.496963 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.513638 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.519580 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.520887 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.525377 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.532658 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.534258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.540315 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:14.559397 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:14.560134 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:14.575302 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:14.587210 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.631464 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.643164 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.646190 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.649838 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.660614 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.663836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.677126 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.689549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.689644 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.707649 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.708504 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:14.726917 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:14.730741 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:14.786593 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:14.790503 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.794826 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.798966 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.799124 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.810511 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.822206 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.836432 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.842007 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.843360 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:14.846726 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.847004 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:14.871809 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.922709 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:14.928035 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:14.930562 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:14.937429 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:14.938899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:14.940820 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:14.947218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:15.014489 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.030620 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:15.045027 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:15.045362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.053416 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:15.054960 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.055930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.057299 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.058940 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.084331 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.084525 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.091271 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.093303 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.095165 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.101177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.105643 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.107895 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:05:15.120414 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:15.138373 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.138988 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.141093 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.142664 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.144523 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.149706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.196977 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:05:15.197603 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:15.257793 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.260499 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.263473 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.268069 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.279716 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.280797 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.281258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.314816 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.319885 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.347458 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.350566 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.355252 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.357174 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:15.363035 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.395627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.402920 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.407758 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.411409 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.417759 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.417937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.424464 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:15.426074 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:15.428916 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:15.434018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.434373 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.458940 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.466069 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.468621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.587365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.595149 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:15.598392 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:15.601291 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:15.609580 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:15.628448 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.633622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.634902 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.648414 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.650066 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.650365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.652135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.653131 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.661960 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.673341 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.682124 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:15.691874 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:15.700570 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.704787 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.717846 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.726433 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.754295 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.754763 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.755095 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.770536 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.781018 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.781602 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:15.784770 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:15.785879 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:15.806931 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:15.810042 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 14:05:15.859618 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.863638 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.866912 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.872318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:15.893726 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.904306 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.943171 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:15.946084 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:15.948462 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:15.956008 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:15.969396 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:15.980760 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:15.997555 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.015360 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.016578 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.020215 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.024123 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.030383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:16.053635 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.055079 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:16.102118 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:16.124106 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:16.128621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.129001 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.134117 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.135576 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.154649 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.170971 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:16.183290 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:16.196338 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.200768 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.207116 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:16.208778 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 14:05:16.210474 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:16.226445 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.238689 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.240411 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.250631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.264098 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.267979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.274782 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.275591 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:16.277238 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.280381 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.289573 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:16.298132 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.299610 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.302183 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.311017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.339223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.341324 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.375921 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.383430 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.386154 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.392525 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.392846 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.409232 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.482996 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.492939 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.494410 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.501338 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.512318 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:16.522811 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:16.530110 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.530944 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.536535 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.538209 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.554086 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:16.566432 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.590718 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:16.604703 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.607874 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.609192 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.620040 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.624332 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.627645 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.628417 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.631438 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.636863 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.663447 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.665066 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.670152 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.670950 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.675738 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:16.706776 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:16.782822 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.788731 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.789391 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.796746 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.798017 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:16.804903 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:16.806010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.812211 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.813228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:16.815269 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:16.823917 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.836329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.836585 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:16.858371 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:16.859586 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:16.975924 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:16.977632 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:16.982107 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:16.991641 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:16.992988 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:16.996517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:17.001307 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:05:17.022488 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:17.023634 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.030854 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.034032 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.036008 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.065936 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.078415 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.111105 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.113662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.119299 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.129002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:17.131408 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:17.138094 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.142210 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.175418 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.178715 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.180346 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.180480 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.191165 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:17.202328 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:17.220367 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:17.226526 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:17.271865 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:17.279357 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.281152 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:17.283756 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:17.290390 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.291769 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.302625 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:17.308843 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:17.343341 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.347656 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:17.354770 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:17.355120 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.364984 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.383600 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.386014 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.393949 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.420038 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.434747 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.445600 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.460566 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.463694 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.475024 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.477161 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.477435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.478365 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.495549 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.496766 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.515631 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.530112 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.532969 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.533794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.536938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:17.542642 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:17.552889 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.555421 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:17.618983 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:17.639979 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.639982 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.644263 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.655439 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.680514 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.684058 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.687526 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.697882 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.698194 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.714021 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.714516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:17.724817 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:17.728811 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.774031 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.776909 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.778630 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.784619 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.793779 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.811620 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:17.814293 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:17.880911 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.892989 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:17.896904 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.926143 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:17.932209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:17.941365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:17.941794 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:17.953121 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:17.953347 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.961966 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:17.966414 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:17.981178 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:17.989885 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:17.999511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.000836 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.008168 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.012398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.013499 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.053889 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.081327 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.081652 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.090653 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.091841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.094185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.097544 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.098367 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:18.110064 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:18.166059 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.168972 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:18.171380 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:18.179148 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.179949 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.181672 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.183185 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:18.224915 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:18.236873 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:18.238832 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:18.250602 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.255081 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.261780 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.262337 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.264061 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.274694 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:18.303169 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:05:18.335147 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 14:05:18.341428 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.344693 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.347687 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.348532 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.351290 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.360455 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.365253 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.369622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.370417 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.381489 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.407397 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.422039 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.482693 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.490897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.494244 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.494887 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.503367 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:18.512672 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:18.513952 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.556423 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.558950 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.564208 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.566900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.582976 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:18.583058 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:18.585278 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:18.592015 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:18.622900 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240502 14:05:18.623581 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 14:05:18.642503 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:05:18.645748 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.651122 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.656502 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:18.663873 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:18.669991 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.677170 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:18.684527 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:18.692541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.737761 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.739053 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.743356 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.749545 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.749891 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.750094 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.824116 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.831378 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.836869 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.846028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:18.847663 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.854984 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:18.861089 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:18.862411 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.869925 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:18.876739 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:18.880729 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.895634 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:18.910100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:18.984650 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:18.990694 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:18.994542 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:18.994688 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:19.002171 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.007982 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.018744 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.027791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.029744 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.031905 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.039902 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.042138 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.049041 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.109915 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.113024 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.128531 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.132213 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:19.137897 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.138685 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.146914 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:19.167609 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:19.180156 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:19.198792 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.207046 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.210575 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:19.217042 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:19.217371 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.218313 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.221060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.226001 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.275760 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.280823 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:19.289309 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:19.292618 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.295003 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.303092 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.320329 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:19.343581 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.386974 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.388876 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.394079 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.403421 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.404099 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.416810 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.419936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.421633 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.427003 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.429116 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:19.429749 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:19.446399 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.458788 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:19.471340 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:05:19.512025 6281 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:19.512158 6281 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:19.512810 6281 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:19.513510 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:05:19.513666 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:05:19.514324 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:19.514432 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:19.514477 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:19.514678 6281 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:05:19.520937 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.521207 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.531980 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.541561 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.554996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.556973 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.563314 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.566776 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.575579 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.575762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.580199 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.598991 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:19.611092 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:19.630940 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.661347 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:19.672521 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:19.676805 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.678881 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.697978 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.703594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.705260 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.712266 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.713011 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.715104 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.734181 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.739097 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.740717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:19.781242 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:19.790199 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:19.796342 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.806883 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.811441 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.814966 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:19.819113 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.833791 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:19.862437 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:19.885113 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.889408 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:19.892637 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:19.913611 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:19.914078 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.918382 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:19.931162 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.975018 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.990005 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:19.990470 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:19.991678 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:19.994794 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:19.994828 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.008443 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.021224 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.035323 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.036593 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.039769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.042475 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.071806 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.111335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.115936 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.121451 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.122395 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.122499 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.128846 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.193179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:20.198918 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:05:20.199179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.206435 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:20.207455 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.212793 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.214967 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.215044 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.260546 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.262286 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.262588 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.265321 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.267910 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.302937 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:20.304028 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:05:20.304843 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.338869 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.340355 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.350569 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.357018 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:20.357507 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:20.376101 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.378962 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:20.388480 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:20.437112 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.439260 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.440246 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.449185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.471560 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:20.486409 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.513471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.516960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.517695 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.517983 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.541283 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.545141 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.559722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.560189 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.565186 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.574911 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:20.578477 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.587081 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.587380 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:20.592994 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:20.600808 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.604646 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.609485 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.612466 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.614961 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.626507 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.630620 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:20.679013 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 14:05:20.695410 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.706621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.709975 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.716006 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.732184 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.735600 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:20.756651 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:20.763010 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:05:20.810091 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 14:05:20.824065 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.825703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.834506 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.845772 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.855440 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.862743 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.871277 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:20.872879 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:20.876781 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.888358 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:20.889750 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.906903 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:20.909101 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:20.919445 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:20.920193 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.921329 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.929239 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:20.929747 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:20.932332 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:20.954406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:20.954753 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:20.978494 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:21.020474 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.028950 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.032635 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.033391 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.037557 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.050567 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:21.095839 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:21.107178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.108565 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.112397 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.116396 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.117616 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.121968 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.169288 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.172292 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.173933 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.174213 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.174517 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.175612 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:21.218631 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:21.283715 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.308729 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.320385 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.330835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.340375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.346621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.356051 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.368008 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.370105 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.371667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:21.375393 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:21.376048 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.384644 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.390949 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.400990 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.403746 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.430405 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.431457 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:21.439419 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:21.444329 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.444921 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.451614 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.459774 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:21.476948 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:21.478488 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.478993 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.479957 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:21.521438 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:21.560611 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:21.573137 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.576761 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.581805 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.583763 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.588719 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.604452 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:21.619120 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:21.620175 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:21.637226 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.652863 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.656177 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.656494 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.664222 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.668241 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:21.676219 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:21.680466 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.685459 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.699201 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.700338 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.704100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.710176 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:21.767635 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:21.829118 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:21.834836 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:21.837663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.837714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:21.841265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.843010 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.852710 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:21.857640 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:21.865396 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:21.941193 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:21.947109 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.947683 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:21.953230 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.963699 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.975134 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:21.985206 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:21.987506 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:21.987892 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.010448 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.018296 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.026531 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.034044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:22.048475 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:22.050036 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.059875 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.065347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.079664 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.097977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:22.104959 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:22.122010 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.123121 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:22.125264 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:22.135140 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.135354 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.140422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:22.159317 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:22.160216 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.192907 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.212462 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.212785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.216853 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.217955 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.225386 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:22.263823 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:22.271708 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:22.287498 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.294890 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.296850 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.310952 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:22.325994 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:22.339344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.344799 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.352494 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.352788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.358597 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.363415 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.365833 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.366778 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.466233 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.468097 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.472916 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.474565 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.476125 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:22.491645 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:22.518816 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.521368 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.527837 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.528486 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.535244 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.536424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.539637 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.616077 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.618140 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.620471 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:22.625451 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:22.631639 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.631982 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.635171 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.638742 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.640915 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.664429 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.665940 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.673326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.682240 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:22.735513 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:22.767441 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:22.823598 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.825899 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.835897 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:22.837694 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:22.840588 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:22.844553 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:22.856001 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:22.858305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.859489 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.868788 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.871728 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:22.881649 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:22.928407 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.937738 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:22.955379 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.959735 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:22.964561 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.967204 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.970830 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:22.975294 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:22.979022 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:22.982290 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.008811 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.018705 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.020954 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.022425 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.036023 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.041335 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.046185 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.053341 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.057332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:23.075100 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:23.083420 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.084184 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:23.084810 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:23.185047 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:23.187183 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:23.189409 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.196905 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.198042 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.213812 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:23.214516 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:23.216454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.224381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.227002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.244617 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.259563 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.261819 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.265043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.265610 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.267505 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.268332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.287003 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.290792 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.294993 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:23.318068 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:23.335460 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:23.410307 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:23.439889 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.444954 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.458398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.468947 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:23.472011 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:23.485020 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.485659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.485666 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.503178 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.505177 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.507086 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.510775 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:23.513767 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 14:05:23.526903 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.529538 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.536193 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.541004 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.543135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.548748 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.562037 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.603215 8598 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:23.603338 8598 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:23.603792 8598 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:23.604489 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:05:23.604635 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:05:23.606057 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:23.606472 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:23.606539 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:23.606674 8598 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:23.630709 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:23.663873 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:05:23.712260 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:05:23.720793 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:23.735041 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:23.735813 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.745520 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.760322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.763839 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.767772 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.781981 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.791697 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.796202 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:23.810256 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:23.810909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.811591 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.813593 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:23.829494 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:23.829813 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.840350 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.841282 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.844069 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.856633 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.858641 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.862708 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.862967 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.867061 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:23.874258 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:23.874809 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.880820 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.884460 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:23.901238 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:23.904785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:23.911403 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:23.926806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:23.927525 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:23.931470 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:23.931803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:23.933089 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:24.058228 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 14:05:24.060428 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:24.079174 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:24.084471 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:24.085273 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.085537 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.086938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.095451 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.099171 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.104712 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.105176 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.115757 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.118958 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.132556 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.141711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:24.197177 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:24.250820 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:24.264987 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.267549 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:24.268863 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:24.274473 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.275640 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.281715 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.291666 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.300628 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.302371 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.306201 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.308173 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:24.345391 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:24.347299 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:24.349654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.353847 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.398587 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.423611 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.429453 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.436751 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.439591 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.441499 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.444974 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.452353 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.459409 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.467233 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:24.474721 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:05:24.513671 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.514544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.524252 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.537446 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:24.537804 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:24.542217 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.553345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.555464 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.563045 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.626628 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.627786 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.629315 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:24.644960 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:24.658746 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.660189 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.667630 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:24.668633 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:24.698396 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.709043 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:24.709161 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.711360 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.722862 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.723371 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:24.768832 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:24.788802 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.792251 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.810132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.813313 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.822620 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.844638 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:24.855887 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:24.868731 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.887727 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:24.895377 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:24.896436 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.897847 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.898057 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.901281 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:24.935289 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:24.945634 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:24.951851 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:24.953760 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:24.969085 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:24.995465 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:24.998940 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.000814 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.002900 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.008410 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.009830 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.028478 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:25.028877 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:25.030174 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.043128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.044528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.051461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.067577 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.076104 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.077229 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.173676 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:25.175365 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:25.180370 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.182480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.185392 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.194947 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.202150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:25.214704 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:25.216116 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.226572 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.229770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.231190 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.234562 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.234828 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:25.265205 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:05:25.316160 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 14:05:25.317931 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.318936 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:25.320428 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:25.331655 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.331801 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.346225 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:25.355029 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:25.364542 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:25.381700 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:25.399484 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 14:05:25.455821 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:25.457015 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.465421 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.467540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.473309 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.502179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.504559 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.507779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.508993 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.519901 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.536633 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.552105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.558987 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.564844 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.574615 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.578495 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.582988 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.586380 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.590387 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.603912 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.604357 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.607918 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.622601 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.623919 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.629119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:25.639374 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:25.709168 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:25.738627 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:25.751298 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.755625 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.762109 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.770130 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.775146 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.778026 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.787974 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.792670 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:25.800992 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:25.801205 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.806890 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:25.807891 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 14:05:25.813060 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.821267 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:25.825870 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.826311 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:25.869886 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:25.870534 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:25.875692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:25.889623 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:25.907264 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:25.980764 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:26.001312 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:26.011117 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.016395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.022426 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.022692 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.033880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.045688 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.046824 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.047185 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.048449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.049610 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.070214 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.085567 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.089159 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.094312 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.109344 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.117993 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.125885 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.131616 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.150267 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.154836 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.164680 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.174619 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.177273 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.179955 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:26.204499 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:26.269866 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:26.287707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.293503 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.301101 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.302667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.309223 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.341367 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:26.370829 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:26.370872 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.371199 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.384186 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.384354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.389025 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:26.404244 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:26.412441 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:26.418071 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:26.453261 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:26.457728 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.458407 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.460328 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.463979 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:26.465471 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:26.480111 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.481112 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.489456 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.499974 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.502507 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.506398 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.514214 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.518342 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.519469 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.527180 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.527741 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:26.540031 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:26.541350 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.550086 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.553141 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.676676 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.677907 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.688423 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.695458 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.697229 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.698336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.699860 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.700335 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:26.710767 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:26.711644 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.722354 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.747081 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.747267 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:26.758561 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:26.801751 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:26.809228 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.815083 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.820993 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.829358 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.832708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:26.838737 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:26.847700 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.850661 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.851650 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:26.861091 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:26.867493 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:26.875948 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.883864 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:26.912786 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:26.954999 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:26.970253 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:26.979197 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:26.981210 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:26.986371 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:26.988973 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.005633 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.010061 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.012857 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.016726 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.019398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:27.019445 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:27.043126 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.057076 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.120600 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.124933 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.129247 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.129262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.129518 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:27.148830 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:27.170686 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:27.181774 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:27.202345 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.208699 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.214501 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.218273 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.219820 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:27.229310 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:27.249758 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:27.265408 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:27.306149 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.320257 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.337165 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.342206 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.346534 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.350236 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.363451 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.371816 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.377920 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.387163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.387766 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.391261 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:27.394203 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:27.398586 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:05:27.426734 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:05:27.449894 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:27.459089 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.472075 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.478111 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.497169 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.503708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.520918 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.524695 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.528360 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.530817 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.554454 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.554567 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.555008 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:27.610816 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:27.629631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.637497 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.640060 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.647401 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:27.655263 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:27.666057 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.671847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.672204 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.705379 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.705677 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.707916 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.708935 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.722087 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.725401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.725750 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.736553 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.744845 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.750806 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.776587 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:27.806272 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:27.823248 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:27.850049 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:27.852110 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:27.856742 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.860054 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.860113 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:27.871270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:27.873620 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:27.893481 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.896055 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:27.901161 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:27.904655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:27.909060 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:27.910281 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.926853 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:27.998340 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.005134 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.012322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.021046 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.023747 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.037277 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.041319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.044984 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.049396 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:28.062685 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:28.064507 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.084983 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.103529 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.106789 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.113929 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.114387 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.123162 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.139973 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.140208 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:28.202517 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:28.228646 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:28.244438 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:28.261263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.261263 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:28.269531 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:28.272670 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.274796 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.278784 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.281669 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:28.326486 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:28.327944 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.339262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.343200 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.344491 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.344748 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.347025 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.359470 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.367763 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.372768 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.373705 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.378578 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.380205 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.405975 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.417213 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.419425 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.423519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.433270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.439710 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:28.448789 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 14:05:28.517467 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 14:05:28.551990 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.555567 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.556872 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.564581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.579326 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.587265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.592904 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.597206 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.597941 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.598703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.614761 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.633867 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.634459 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.640452 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:28.652777 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:28.667130 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.671698 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.674970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.682621 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:28.750010 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:05:28.769062 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.770695 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.776000 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.781419 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.781466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:28.800555 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:05:28.801908 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:05:28.812121 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:28.820147 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:28.834919 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.843860 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.845753 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:28.847847 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:28.849133 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:28.849433 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:28.864257 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:28.882735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.884384 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:28.886529 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:28.897449 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:28.900204 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:28.908716 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:28.909956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:28.920753 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.032429 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.035552 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.038465 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.057631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.061535 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.061851 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.070911 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.077095 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:29.080410 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.092214 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:29.098011 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.105278 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.118402 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.125082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.133235 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.137261 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.154403 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.156383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.157779 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.169409 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:29.171510 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.209466 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.210003 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.211028 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.219504 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.220755 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.234853 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:29.251819 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:29.322319 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:29.340158 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:29.353804 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:29.358387 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.367673 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.369248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.374861 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.376485 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.377372 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.396858 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:29.409157 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.409408 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.428980 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.429098 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:29.429344 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.441195 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:29.452334 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:29.458602 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:29.472651 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.474706 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.475977 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.476173 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.483482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.485670 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.550257 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.552678 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.559376 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.564742 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.575294 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.575575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:29.577152 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.614717 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.619630 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.623927 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.624222 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.624722 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.632464 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.694219 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.695917 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.696264 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.707180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.717693 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.718293 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:29.742381 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:29.744031 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 14:05:29.756366 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:29.772230 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:29.781075 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.781668 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.783432 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.790905 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.791891 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:29.792260 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:29.799026 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:29.835763 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:29.871006 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.877660 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.879236 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.883725 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.887913 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:29.898464 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:29.905838 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.919817 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.941622 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:29.944401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:29.946756 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.950352 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:29.957194 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.964505 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.964519 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:29.992022 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:29.993639 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:29.998401 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.007032 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.115798 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.124544 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.125206 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.140789 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.145871 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.150929 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.171299 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.174559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:30.175292 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:30.178902 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.191387 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.194914 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.206377 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.207746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.211355 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.216441 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.218878 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.219321 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.226939 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.228889 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.232403 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.268503 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.273571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.283589 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.284521 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.295732 11342 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:30.295852 11342 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:30.296253 11342 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:30.296942 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:05:30.297081 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:05:30.305770 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:30.309360 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:30.322208 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:30.322644 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:30.322736 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:30.322860 11342 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:30.334333 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:30.379324 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.380362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:30.382531 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:30.385052 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.385449 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.394985 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:30.400362 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:05:30.411288 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:30.456863 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 14:05:30.458613 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.459199 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.463573 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.463698 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.476814 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.489531 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:30.498747 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:05:30.515131 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:30.528492 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.529716 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.531006 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:30.534807 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:30.539973 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.543658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.544373 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.641976 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.646737 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.652101 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.656407 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.675566 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.679569 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.715519 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:30.716856 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:30.717060 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.726624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.735592 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.750845 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.755618 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.756402 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.772769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.773180 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.773725 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.774259 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.783937 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:30.841934 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:30.861891 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.875463 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.875623 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.876107 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:30.892024 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.897293 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:30.909880 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 14:05:30.911307 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:30.915036 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:30.940485 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:30.946918 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:30.963634 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:30.970309 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:30.972088 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:30.978785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.037029 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.039788 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.050988 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.052232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.056424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.078985 30896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:31.079044 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 14:05:31.080818 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.086027 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.097805 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.101573 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.118202 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.120571 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.124635 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.127722 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.129878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.135903 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:31.136798 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.144620 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.169312 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.203622 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.209023 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:31.213754 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:31.215314 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.221470 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.221884 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:31.235532 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:31.237128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.309391 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.316601 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.316666 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:31.318466 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:31.335096 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.335377 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.352015 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.382268 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.387694 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.389488 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.389688 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.391232 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.394594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:31.426465 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:31.469873 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:31.487179 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.488363 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.494851 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:31.497221 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.498466 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.509446 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.526624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.543298 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.551887 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.553313 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.560317 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.561443 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.568410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:31.576309 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.585247 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.616501 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.620004 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.623255 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.635227 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.645047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.647455 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.739260 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.747166 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.751845 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.753185 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.762408 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:31.770903 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:31.790448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.794363 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.804414 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:31.812924 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.816109 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.819542 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.836576 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:31.846010 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:31.847440 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.879299 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:31.887943 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:31.896574 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:31.897143 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:31.897810 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.899525 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.900091 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:31.907419 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:31.908066 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:31.909108 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.919263 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.920544 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.921034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.935590 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.957216 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:31.967744 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:31.977532 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:31.985527 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:31.992460 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:31.999199 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:31.999575 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.001749 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:32.049647 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:32.074189 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.079115 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:32.083375 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:32.093065 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.093428 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.104213 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:32.108551 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:32.117367 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.143599 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.147670 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.147819 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.151671 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:32.161223 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:32.162326 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.166798 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.240418 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.243484 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.247779 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.248456 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.249194 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.252897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:32.293273 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:32.306991 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:32.310328 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.313402 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.319109 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.319603 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.330372 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.339816 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.418576 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.429072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.433189 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.455637 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.456688 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.472043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.475543 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.485535 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.493197 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.500834 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.516151 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.531664 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.533614 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.544166 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.555701 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:32.568210 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:32.587154 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.587515 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.600986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:32.614578 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:32.620472 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.630892 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:32.639410 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:32.642146 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.647313 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.656730 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.657052 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:32.675521 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:32.677371 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:32.695991 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.698261 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.707372 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.708847 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.717200 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:32.731758 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:32.739464 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.746598 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.747088 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.747821 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.755630 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.757148 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:32.765975 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:32.781148 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:32.800616 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:32.831027 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.837236 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.838881 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.842612 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.843151 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:32.844483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:32.859664 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:32.941694 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:32.947736 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:32.948127 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:32.955736 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:32.957497 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:32.969458 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.023636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.029949 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.033555 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.040568 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.046228 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.047452 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.050194 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:33.056527 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.059046 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.072393 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.077279 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.086066 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:33.086748 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:33.097213 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.177800 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.178575 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.192790 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.197103 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.199517 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.200896 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.202227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.211128 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:33.218051 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:33.229370 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.236450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.236932 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.246933 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:33.261871 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.290828 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.300084 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.325477 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.349977 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.358847 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:33.371517 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:33.372701 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.374034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.378712 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:33.389261 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 14:05:33.390852 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.398900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.402019 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.407917 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:33.446723 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:33.468725 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:33.484722 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 14:05:33.527269 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.528788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.529716 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.538995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.540292 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.542930 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.547504 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.549301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.565294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.580902 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.582676 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.588083 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.590695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.621274 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.633150 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.648595 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:33.650954 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.651392 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.654338 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.655467 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.658794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.662247 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.662348 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:33.671618 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.675452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.705015 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.742632 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.746608 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.759889 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.760138 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:33.760847 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 14:05:33.762871 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.772567 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:33.777853 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:33.820884 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:33.825572 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:05:33.846153 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:33.850795 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.859315 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.862092 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:33.868876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.873287 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:33.910410 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.911423 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:33.912358 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:33.917266 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:33.924579 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:33.925956 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:33.928411 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:33.930652 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:33.986739 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:34.033452 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.039429 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.040810 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.046319 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.057267 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.080829 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.099400 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.099725 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.102543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.113246 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:34.114243 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:34.133735 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.147321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.150084 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.153484 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.155155 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.165321 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.186187 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:34.192010 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:34.197419 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:34.247007 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:34.289222 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.291115 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.291402 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.294677 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.334025 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.346371 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:34.348985 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:34.353013 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.364053 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:34.365658 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:34.366241 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.366853 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.377135 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:34.391271 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:34.397274 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.398223 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.399158 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.405299 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.413410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:34.423920 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:34.432072 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:34.437068 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:34.442974 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:34.450826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.474844 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.478932 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.479743 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.490743 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.493213 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.523407 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.527734 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.528746 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:34.534796 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:34.540112 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.553752 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.558820 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:34.568733 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:34.571246 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.616672 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.621793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.632489 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.635550 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.655115 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.655921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.718379 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.728595 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.729621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:34.733363 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:34.746186 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.751863 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.758935 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.800196 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.801087 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:34.811997 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:05:34.815912 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.828714 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.832864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.849416 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:34.860494 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:34.863164 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:34.871979 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.879166 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.881832 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.885951 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.914268 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.952054 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:34.955061 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:34.961273 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:34.969390 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:34.976922 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:34.983949 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:34.987025 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:34.990491 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.001686 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.006364 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.010010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.010318 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.024466 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:35.071180 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:35.117497 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:35.133452 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:35.134792 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.148869 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.153720 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.154171 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.160748 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.164011 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.166256 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.169996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.180724 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.190076 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.190919 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.254895 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:35.257584 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:35.262928 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.268774 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.274217 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.296591 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.300843 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.313510 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:35.315449 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:35.320380 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.321187 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.333181 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.339257 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.343268 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.344965 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.350997 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:35.380611 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:05:35.384358 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:35.384697 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:35.393358 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:35.393901 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.398339 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.404568 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.452919 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.453974 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:35.456393 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:35.465538 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.465694 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.475284 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.486933 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.537734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.538369 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:35.540606 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:05:35.543133 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.545481 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.548513 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.555269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.604455 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.606719 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.606830 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.609194 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.609870 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.612769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:35.654028 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 14:05:35.666093 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 14:05:35.676856 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:35.684954 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:35.706621 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:35.726802 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.733129 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.734342 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.749063 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.755198 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.766374 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.770664 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:35.781564 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:35.786073 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.787050 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.791436 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.792448 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.809973 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.814508 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.827222 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.830179 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:35.832324 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:35.832628 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:35.838550 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:35.867509 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:35.973714 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 14:05:35.976833 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:35.981985 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:35.983089 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:35.996222 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.001942 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.011412 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:36.016445 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:36.031914 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.039981 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.048846 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.052397 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.058473 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.063097 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.079089 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.079322 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.083307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.084437 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.089109 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.098934 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.104496 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.116966 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.118012 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.118239 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.120585 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:36.126145 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.127437 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.135799 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:36.196630 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:36.267359 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.276157 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.283735 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.284377 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.292461 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.304690 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.308063 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.326992 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:36.336884 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:05:36.341861 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.363934 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.365090 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.373798 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.373967 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.387598 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.388821 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.395628 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.396894 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.400194 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.404171 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.410813 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.414959 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.421715 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.453558 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.460240 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.461993 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.470988 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:36.542984 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:05:36.585690 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 14:05:36.592762 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:36.606900 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.607791 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.630962 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.634306 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.640223 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.650614 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.651335 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.654194 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.660845 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.666214 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.667773 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.669129 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.677883 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.679047 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.679602 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.683338 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.688655 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.688709 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:36.689896 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.700978 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.701031 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.708499 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.709538 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.714347 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.753233 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:36.800716 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:05:36.804908 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:36.846154 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:36.861337 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.862944 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.886703 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:36.890913 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.891530 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:36.894939 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:36.915215 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:36.915946 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:36.918459 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:36.918663 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:36.922974 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:36.938300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:36.940515 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:36.942808 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.007205 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.007891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.009212 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:37.017552 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:37.025398 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.031155 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.031419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:37.040335 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:37.121141 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.130427 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.134510 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.138170 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.141419 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.141996 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.148818 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.153250 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.188733 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.190510 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.192821 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:37.198031 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:37.199038 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.248972 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.258688 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.266853 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.268520 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.278216 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.280241 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.282840 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.297443 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.312817 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.313320 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:37.317577 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:37.319276 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.321117 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.408741 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:37.409646 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:37.410176 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.411788 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.412879 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.426975 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.437826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.444895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.459530 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.462615 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.463629 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.467389 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.482033 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.532605 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.533895 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.536799 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.541183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.563153 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.563503 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:37.634320 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:37.638626 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:37.650473 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.661581 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.662135 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.670625 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:37.672719 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:37.679301 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.684490 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.689399 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.704464 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.706265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:37.710103 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.714031 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.714205 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.738734 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:37.740180 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.746500 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.753325 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.759433 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.767423 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:37.785634 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:37.823968 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:05:37.825525 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:37.831893 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:37.836544 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:37.840134 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.840538 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.842031 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.843549 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.874212 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:37.897812 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:37.899720 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:37.918663 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:37.922483 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:37.927392 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:37.940603 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:37.943149 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:37.965862 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.022073 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.024053 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.038309 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.044265 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.048763 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.049360 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.051551 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.070190 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.072788 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.073751 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.082845 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.093955 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.095808 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.097419 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.108161 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.111274 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:38.115620 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.122933 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.125156 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:38.140542 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:05:38.155112 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:38.173635 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:38.176836 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 14:05:38.213027 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.235574 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.240926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.241194 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.253012 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.264300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.269300 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.306663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.306758 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.321897 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.326486 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.329028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.355121 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.385897 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.401364 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.401443 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.404100 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.426463 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.429339 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.429658 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.429859 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.435669 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.443265 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.463984 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.472554 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:38.523420 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:38.562229 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.563478 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.570030 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:38.572474 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:38.584141 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:38.593676 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.599103 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.599694 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:38.612859 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:38.614631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.617034 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.619601 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.630342 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:38.635160 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.641383 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:38.657125 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:38.661623 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.670217 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.677543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.681007 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.705021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.705032 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.713348 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.716821 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.726876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.732254 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.735196 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.748803 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:38.763502 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:38.789166 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240502 14:05:38.792613 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.834683 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.840834 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.841054 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.841245 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.854823 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.876787 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:38.893031 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:05:38.893654 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.948417 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:38.949609 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.949842 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:38.950516 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.958192 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.961349 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:38.976992 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:38.980594 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:38.985072 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:38.990985 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:38.997095 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.004531 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.005106 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.009541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.012133 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.013775 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.020025 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:39.031170 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:39.039767 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.052492 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:39.087797 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:39.128607 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.145365 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.145977 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.150099 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.154712 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.158691 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:39.203562 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:39.208786 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:39.209167 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:39.214725 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.223594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.232553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.233950 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.241029 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:39.296612 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:39.298347 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.308851 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.333976 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:39.337074 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:39.337580 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.338300 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.344729 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.373608 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:39.380069 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:39.386576 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.393956 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.403399 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.405107 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.407780 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.416750 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.417613 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.431618 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.443372 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:39.449925 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:05:39.452041 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:39.458487 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.460160 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:39.495656 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:39.510826 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.511353 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.520187 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.520207 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.522362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.540498 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:39.547910 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:39.547986 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:39.604405 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.617005 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.624455 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.624706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.627008 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.632997 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.688784 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.697202 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.700871 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.706802 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.710309 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.717258 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.765295 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.776979 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.778532 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.778851 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.791663 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.824847 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:39.845180 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:39.854254 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:39.877768 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.882782 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.892318 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:39.893937 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.893957 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.894896 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:39.897035 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.911294 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.912155 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.919346 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.921079 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:39.921909 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:39.957224 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:39.976526 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:39.985901 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:39.988432 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:39.989614 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:39.991482 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:40.009286 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:40.015991 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.022692 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:40.050544 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:40.079149 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.086549 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.096181 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.109355 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.116988 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.126518 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.151367 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.158494 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.169760 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.175730 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.184662 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.201712 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.203318 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:40.206745 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:40.207217 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.216753 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.217026 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:40.220044 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:40.221143 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:40.231146 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:40.241209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.255149 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.270033 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.279489 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.284698 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.305003 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.307225 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.367817 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.367944 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.368176 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.379184 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.383472 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.388517 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:40.388669 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:40.403635 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:40.414781 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.427737 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.430207 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.430403 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.444779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.453276 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.476601 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.490633 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.493199 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.494897 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.496716 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.499714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:40.506323 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:40.518394 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:05:40.594698 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:40.600243 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:40.607731 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:05:40.609771 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:40.612419 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:40.620759 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.620932 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.630056 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.645650 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.662681 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.662777 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:40.662914 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 14:05:40.666800 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.690294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.708703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.710012 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.718335 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.722920 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.723726 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.736860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.740247 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:40.746258 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:40.749173 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.757630 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.765475 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.767199 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:40.777060 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.778429 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.781311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.902427 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:40.907471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.908051 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.915207 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.932329 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.936656 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:40.951539 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:40.969884 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:40.970841 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.970929 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:40.979293 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:40.983697 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:40.983920 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:40.987562 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:40.991179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:40.994621 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:05:41.005031 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.005025 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.006752 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:41.012048 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:41.012530 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.030742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:41.050374 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 14:05:41.090740 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:41.109606 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:41.159888 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.160799 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.162338 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.168658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.169739 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.180927 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.185896 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.188184 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.189911 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.195772 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.198742 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.208887 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.211380 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.211735 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.216440 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.217656 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.223428 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.269176 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:41.374565 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:41.377655 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:41.380795 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:41.393054 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.398403 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.408381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.410977 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.412019 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.420444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.426278 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.426748 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.428886 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.450870 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.467048 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.473330 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.481177 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.493197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.500972 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.501570 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.501909 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:41.503564 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:41.510831 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.518359 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.520627 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.522131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.532974 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.535318 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.537847 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:41.581940 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:41.628561 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:41.699800 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:41.710170 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:41.713467 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.719055 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.724292 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.727587 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.764847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.768342 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.775100 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.776111 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.791735 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.796821 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.802917 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.811554 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.813117 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.820971 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.837762 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.838835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.839830 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:41.840373 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:41.842638 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:41.846463 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:41.858306 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:41.858775 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:41.860422 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:41.869020 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:41.879184 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.882776 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.891327 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.913784 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:41.916671 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:41.921281 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:41.926939 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:41.956974 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:41.969352 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:42.000885 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:42.050962 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.066823 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.068639 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.077322 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.078931 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.079556 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.081624 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.085865 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.094782 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.099174 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:42.099365 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:42.142105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.146412 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.163733 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.172322 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.176998 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.180022 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:42.181870 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:42.191787 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:42.204020 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:42.204380 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:42.259707 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:42.280231 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.284600 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.285180 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.292395 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.299127 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:42.321341 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:42.328532 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.340384 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.354893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.358460 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.360296 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.379230 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.389850 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.399920 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.405500 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.408763 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.417826 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.424435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.427807 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:42.440845 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:42.477509 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:42.503803 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.506496 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.507017 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.509529 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.523350 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.533807 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.589958 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.591288 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.597044 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.603720 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.605144 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.611105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.611095 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.621122 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.640659 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.646522 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:42.654362 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:42.665287 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.668519 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:42.690924 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:42.702037 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:42.707821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.713459 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.724813 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.727391 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.730208 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.752923 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:42.771899 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:42.792924 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.796466 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.806852 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.822207 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.824872 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.852425 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.857455 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.895684 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.898429 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.902674 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:42.904075 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:42.905658 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:42.912211 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.914496 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:42.915016 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:42.917986 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:42.923904 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:42.929270 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:42.930774 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:42.936115 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:42.940848 18064 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:42.940946 18064 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:42.941382 18064 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:42.941879 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:05:42.941999 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:05:42.943821 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:42.943907 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:42.943938 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:42.944200 18064 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:42.983341 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:43.047914 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:43.059952 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:43.076508 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.076515 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.086295 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.092499 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.102818 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.106596 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:43.113830 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:43.120003 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.123076 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.126238 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.145283 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:43.156330 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:43.158244 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.163897 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.171651 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.177995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.188949 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.190680 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.208897 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.212234 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.269172 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:43.277693 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:43.282718 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.295683 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.299139 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.302628 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.314145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.364933 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.365082 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.369419 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.371156 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.412137 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.418154 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:43.418752 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:43.427443 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:43.433197 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.434011 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.434536 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.435775 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.437906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.438350 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.441756 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.442153 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.448930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.451046 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.472950 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.483738 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:43.511437 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 14:05:43.553848 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.555721 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.556066 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.563079 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.577404 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.593770 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:43.606400 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:43.620200 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:43.653055 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.654376 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.654510 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.659709 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:43.661494 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 14:05:43.679785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.692287 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.701634 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.704677 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:43.705454 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:43.707702 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:43.708225 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.709726 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.712893 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.730433 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.767354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.771476 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.774950 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.778522 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.779311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.793920 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:43.846550 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:43.864162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.864830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:43.867769 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
I20240502 14:05:43.869081 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.869904 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.870121 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:43.894531 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:43.900512 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:43.903489 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:43.923650 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:05:43.947080 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:43.950744 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:43.964118 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:43.968690 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:43.969305 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:43.970629 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.050179 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.051391 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.065201 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.073190 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.074719 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.079835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.111841 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.120420 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.120679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.124250 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:44.129442 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:44.147025 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.158349 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:44.170933 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:44.177472 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.186112 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.190287 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.192515 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.199354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.200117 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.201754 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:44.213335 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:44.242841 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.243553 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.245891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.249084 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.266535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:44.298044 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:44.320448 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:44.391302 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.396216 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.412204 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.412387 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.412779 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:44.432544 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:44.432787 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.437280 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.442286 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.443526 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.446599 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.468366 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.480515 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.498471 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.500087 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.501895 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.503636 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.510028 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.515794 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.528740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:44.528847 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:44.531018 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.533345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.544859 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.565491 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.592016 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.597424 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.598060 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.615489 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.615727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:44.618606 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:44.622948 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.657832 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:44.682736 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:05:44.684859 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:05:44.714015 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:44.721379 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:44.728677 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.735882 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.740773 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.751272 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.766855 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.770778 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.802307 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.814941 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.815168 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.816269 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.825274 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.837513 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:44.853008 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:05:44.868994 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:44.869124 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:44.873404 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:44.889917 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.896190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:44.897049 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:44.900915 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:44.907194 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.993662 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:44.996531 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:44.998767 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.012259 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.016469 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.031916 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.073457 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.085610 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.089190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.089794 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.092152 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.098542 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.119190 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.122690 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.125525 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.127187 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.133556 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.140694 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.160439 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:45.163524 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:45.169641 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:45.176285 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:45.188947 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:45.189267 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:45.192662 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.194504 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:45.199180 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:45.213196 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:45.268816 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:45.282310 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:45.289885 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.292667 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.292922 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.319664 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.321398 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:45.321628 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:45.332727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.364979 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.366143 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.366410 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.380728 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.382087 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:45.410729 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:45.421984 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.429448 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:45.434543 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:45.436159 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.447929 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.454224 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.454916 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.456473 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.462407 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.470088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.475539 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.479329 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.488730 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.493331 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.569816 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.572525 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.578152 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.583248 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.590174 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.591365 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:45.603847 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:45.666538 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.667377 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.668409 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.670894 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.681402 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.685156 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.691928 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.698210 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.706287 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.716964 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.733314 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.736717 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:45.737149 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:05:45.737759 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:45.804328 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 14:05:45.815521 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:45.843158 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.858549 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.863215 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:45.885084 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:45.887446 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.890069 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.894433 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:45.894604 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:45.898861 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:45.909039 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.914536 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.915606 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.916453 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:45.919667 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.920646 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.928340 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:45.937045 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:45.938118 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:45.952162 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:45.983453 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:46.049352 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 14:05:46.061553 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.084163 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:46.111747 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:46.112025 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:46.122186 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.133430 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:46.146950 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 14:05:46.147617 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.151450 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:46.169514 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:46.171790 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.177983 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.178675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.200160 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.204178 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.211980 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.214453 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.216442 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.218003 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.222407 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.223330 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.232790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:46.265967 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:46.267127 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.275844 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.277580 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.289120 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.292403 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.295437 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.317256 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.325654 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.326159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.335088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:46.343715 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.345264 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.356642 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.357915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.365543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.453645 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.470724 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.473116 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.473340 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.487776 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.497267 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.504951 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.512248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.516294 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.522718 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:46.534319 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.548069 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.550873 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.597030 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.601123 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.603823 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.606278 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.608633 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.609524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.677628 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.682152 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.691120 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.693266 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.695209 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.705077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:46.756217 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:46.759352 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.762034 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.762770 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.767647 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:46.779017 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:46.781033 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:46.784624 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:46.812307 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.822329 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.830997 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.831240 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:46.842525 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:46.844158 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.852064 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:46.857177 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:46.960439 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:46.970853 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:46.976176 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:46.990609 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:46.990617 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:46.999338 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.009193 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.028228 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.028957 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.034802 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.043587 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:47.044855 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:47.058454 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.067752 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.082818 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.094822 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.099838 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:47.101248 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:47.112744 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.117589 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.123960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.140305 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.141701 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.147183 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.157413 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:47.158197 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:47.172672 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.183315 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:47.193814 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:47.197244 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.204851 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.210773 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.211429 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.227340 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.239552 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:47.254207 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:47.258525 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:47.317301 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.331069 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.345063 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.347024 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.360685 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.363059 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.367789 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.367903 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.375025 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.375736 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.376438 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.379053 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:47.387372 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:47.408831 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:47.449400 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.450911 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.452126 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.453217 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.456880 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.462070 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.523280 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.528290 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.531013 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.535857 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.541488 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.549983 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:47.569772 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:47.604403 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.606561 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.611734 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.614849 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.633175 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.636969 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:47.676664 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:47.678388 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:47.684612 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.686183 22035 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:47.686295 22035 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:47.686841 22035 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:47.687500 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:05:47.687639 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:05:47.691310 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:47.691438 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 14:05:47.692024 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:47.692121 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:47.692153 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:47.694404 22035 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:47.699740 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:47.704406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.707865 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.718371 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:47.718569 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:47.727864 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:47.738972 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:47.746975 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:47.790854 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.801519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.802796 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.807843 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.808655 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.809006 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.879427 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.879813 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.880159 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:47.885286 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:47.898025 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.902670 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:47.912254 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:47.912802 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:47.932641 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:47.935871 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:47.945699 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:47.950657 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:47.957089 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:47.972652 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:47.991287 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 14:05:48.005563 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.005837 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.017095 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.021008 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.024170 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.032146 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.032164 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.032740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.033668 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.034766 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.034790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.035002 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:48.120832 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:48.166180 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.166357 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.166594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.167749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.171613 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.192389 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.242077 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.250690 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.252079 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.253233 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.256814 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.258639 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.260022 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.268929 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.274646 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.275978 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.281687 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:48.283733 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:48.289279 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:48.323412 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:48.323544 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:48.339937 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:48.402339 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:48.419364 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.419566 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.424597 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.436808 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.444444 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.474155 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.475620 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.484737 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.492677 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.498919 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.499518 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.522370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.525688 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.543480 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.560534 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.575250 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.588397 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.592787 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.597196 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.604646 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:48.605201 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:48.614816 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.630802 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.634013 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.643318 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.705124 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.711174 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.714586 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.716502 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.727041 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.732578 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:48.764941 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:48.768378 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:48.782286 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:48.784588 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:48.791270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:48.792654 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:48.794217 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:48.807863 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.811282 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.822403 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.824476 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.827030 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.828933 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.830265 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.830993 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.839854 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:48.847594 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:48.895422 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:48.909154 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240502 14:05:48.909806 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:48.937618 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:48.941778 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:48.944502 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:48.953254 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:48.955777 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:48.965242 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.013034 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.018667 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.021260 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.024183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.024482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.030658 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:49.061043 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 75 similar messages]
I20240502 14:05:49.091493 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.093473 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.099325 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.107095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.111797 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.119705 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.146883 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:49.148380 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:49.158639 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.160535 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.185343 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.185433 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.197571 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:49.200181 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:49.204047 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.206884 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.216446 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.220171 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.233048 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:49.234690 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:49.234913 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:49.306986 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:49.327952 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.348049 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.355870 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.362896 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.370692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.378544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.393679 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.402546 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.409803 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.422014 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.437307 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.449209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:49.449537 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:49.469791 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:49.483098 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:49.501376 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:49.508102 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.510015 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.516075 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.517308 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.518150 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.519101 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.523728 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.533371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.558234 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.562218 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.572093 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.657979 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.667117 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:49.671317 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:49.678606 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:49.682247 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:49.690274 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.691817 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.711836 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:49.720806 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:49.721292 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.725675 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.730170 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:49.733826 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:49.738252 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.744848 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.748756 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.754734 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.764655 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.772159 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.777441 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.792574 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.797727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:49.813331 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:49.901517 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.920817 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.923162 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.927670 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:49.940835 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 14:05:49.942634 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:49.947593 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.948136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.959261 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.966364 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:49.968154 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:49.970275 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.974867 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.975333 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:49.987473 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:49.988145 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:49.992996 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:49.993453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.010151 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:50.017659 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:50.034598 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.058197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.074218 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:50.089645 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 14:05:50.124415 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:50.150048 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.150980 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.156948 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.157442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.165823 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 14:05:50.170869 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.195662 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.229506 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.231395 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.232162 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.240640 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.253974 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:50.257612 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.260237 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.263790 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.264331 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.272913 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.278169 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.281884 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.284346 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.284569 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.296921 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.310428 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:50.317823 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:50.318730 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:50.324887 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.328060 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:50.336136 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.337703 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.341174 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.342411 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.359597 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.359932 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.378309 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.378785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:50.438393 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:50.520457 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.526636 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.529309 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.540906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.541172 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:50.551327 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:50.568830 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.589218 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.601657 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.606837 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.614373 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.615345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.618189 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.630084 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.632508 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.633293 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.641047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.645465 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.664207 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.665002 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.676060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.679198 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.692425 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.701596 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.712615 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:50.799357 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:50.828936 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.833798 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:50.836177 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:05:50.841395 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.841449 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.852027 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.853588 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.858345 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.874701 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.877305 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:50.878111 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:50.888255 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:50.890995 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:50.897457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.914680 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:50.925283 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:50.937616 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:50.940124 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:50.941900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:50.943607 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:50.967362 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:51.010493 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:51.058656 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.067855 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.073520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.080072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.084545 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.089543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.089766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.096904 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.105155 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.112257 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:51.112759 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.121928 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.123109 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:51.140858 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:51.142185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.143819 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:51.144439 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.147471 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.156080 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.158291 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.168494 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.181747 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:51.260023 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:51.299499 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.311195 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.312207 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.323179 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.329183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.338354 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.348037 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:51.349479 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:05:51.363548 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.369426 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.375281 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.380059 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.381824 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.385452 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.399821 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.400563 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.406234 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.407083 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:51.423719 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.431901 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.461660 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.463649 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.469053 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.470921 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:51.475225 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:51.476873 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.489275 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:51.501881 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:51.502455 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.569772 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.578159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.584398 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.585250 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.599741 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.629417 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.651412 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.653515 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.663189 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.663630 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:51.671794 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:51.676864 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.681624 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.687201 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.691805 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.697975 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.698618 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.699674 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.714617 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:51.789824 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:51.865269 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:51.873191 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:51.875793 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 14:05:51.882465 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:51.883008 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.890862 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.898188 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.901242 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.928171 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.929090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.949173 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:51.954569 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:51.954844 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.955363 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.972488 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:51.974036 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:51.974639 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:51.979308 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.979681 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:51.980086 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:51.992918 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:51.994869 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:51.995481 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:51.998811 24054 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:51.998907 24054 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:51.999295 24054 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:52.000442 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:05:52.000597 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:05:52.005043 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:52.005174 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:52.005223 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:52.005448 24054 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:05:52.006397 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.007057 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.014693 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.028532 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.029912 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.032574 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.042546 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.042788 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.060590 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.062769 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.073168 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.086885 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:52.093464 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:52.185457 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:52.222266 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.234162 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:52.242569 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:52.248358 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:52.253414 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:52.257244 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.258016 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.271553 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.278100 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.284425 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.287593 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.288918 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:52.298295 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:52.300114 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.302266 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.321151 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:52.354545 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:52.357488 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.357599 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.360216 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.370302 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.372457 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.374285 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.381337 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.385114 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.388139 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.392175 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.410637 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:52.451718 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:52.477583 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:52.491120 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.492113 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.492388 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.500483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.500715 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.502540 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.575063 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.576135 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.586205 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.598699 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.607154 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.607697 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:52.626091 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:52.658744 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.659847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:52.669682 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:52.670305 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.674314 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.687458 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.707679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:52.723917 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:52.765842 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.773875 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.776084 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:52.786655 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:52.787945 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.792302 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:52.795130 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:52.795841 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.821233 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:52.824177 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:52.828531 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.830520 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:52.832566 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:52.833750 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.861872 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:52.881453 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:52.887284 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:52.890620 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:52.894502 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:52.900854 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:52.902143 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 14:05:52.907114 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:52.914709 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:52.963312 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:53.014015 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.022814 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.025189 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.032934 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.035496 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.047303 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.113335 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.121109 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.130626 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.132246 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.142344 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:53.148999 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:53.156826 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.158074 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.159440 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.161593 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.169804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.181766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.199829 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:53.203123 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:53.208971 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.211577 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.218158 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.228264 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.240476 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.248039 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.248301 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.250675 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.263438 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.276085 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.284499 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.289052 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:53.333345 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:53.391300 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.395087 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.402575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.411860 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.418300 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.435819 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:53.438989 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:53.452494 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.466077 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.479197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.486582 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.495570 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.504555 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.504859 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.506417 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.507939 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.514701 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.517225 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.518546 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:53.522284 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:53.522945 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.534564 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.539952 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:53.556789 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:53.566085 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.581319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.591674 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:53.604355 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:05:53.651512 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.657708 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.659242 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.662714 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.687638 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.689989 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:53.737972 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:53.754419 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.765448 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.767972 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.769263 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.770694 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.796782 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:53.804349 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.808455 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.808530 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.827323 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.828127 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:53.838122 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:53.861966 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:53.875190 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:53.908994 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:53.913764 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240502 14:05:53.914376 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:53.926358 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.931345 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:53.931370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:53.944869 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:53.952984 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:53.953996 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:53.955358 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.986495 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:53.991909 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:53.995887 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.001088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.008723 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.012540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:54.053478 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:54.057569 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:54.067757 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.068184 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.071041 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.075405 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.083424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:54.126956 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:05:54.130218 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.157058 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.158380 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.166280 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.169793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.171319 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.179008 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:54.189961 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:54.245159 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.250064 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.250559 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.260007 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.272459 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:54.275068 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:54.289242 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.290858 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.301334 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.314527 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.319528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.326287 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.336205 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.437494 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.441217 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:54.443015 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:54.447690 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:54.448006 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:54.456928 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.459793 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:54.461199 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:54.470254 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.479251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.479867 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.490345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.494623 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:54.511384 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:54.511871 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.537456 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.551369 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.554082 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.561722 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.574132 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.574935 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.576377 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.577674 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.577973 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.582823 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.591133 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.596266 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.609053 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:54.611053 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:54.642964 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:54.678336 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.680972 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.691802 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.693799 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.705615 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.709367 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.761320 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.765475 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.776876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.778921 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.789878 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.791383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:54.798719 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:54.836045 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:54.842613 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:54.889703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.893707 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.917480 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.931622 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:54.933490 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:54.951114 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.959506 31162 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:54.960422 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:54.961004 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:54.961354 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:54.971927 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:54.977098 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:54.992362 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.012848 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.017086 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.018501 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.019809 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:55.033124 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:55.036424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.036967 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:55.048624 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:55.073228 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:55.102099 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:05:55.125448 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.129149 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.131897 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.138373 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.152982 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.182829 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:55.202100 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:55.218746 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:55.219946 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.228246 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.233973 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:55.234032 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:55.242286 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.244104 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.245270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.249859 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.250134 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.256439 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.261385 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:55.263525 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:55.271651 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.284111 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:55.326400 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:55.332943 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:55.334394 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:55.334551 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.343921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.362123 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.366295 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.374302 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.385378 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.387463 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.391696 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.399122 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.400470 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.402326 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.492707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.495429 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.500761 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.503521 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.513031 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.515031 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.550999 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:55.558358 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 14:05:55.564091 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.564461 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.573388 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:55.575419 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:55.576939 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:55.584388 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.593984 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:55.611843 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:55.635661 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.637035 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.640444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.653347 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.668025 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.670161 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:55.683351 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:55.739303 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.743083 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.749540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.753582 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.758221 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.792491 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.800127 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.801215 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.802762 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.807358 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:55.816395 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.818816 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.840509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:55.846231 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:55.855732 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:55.867523 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:55.873124 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:55.893913 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:55.899627 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 14:05:55.966416 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:05:55.996320 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.003507 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.044517 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:56.044538 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.053081 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.053105 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.057556 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.062229 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.069814 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:56.078734 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.086278 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.094444 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.101660 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.116411 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.124020 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.133113 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.137881 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.139986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.153657 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.158885 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:56.162427 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.165668 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.184033 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.202116 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.211472 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.216352 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:56.216871 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:56.216979 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:56.219185 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.250916 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.267939 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:56.274302 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.281867 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.285567 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.292259 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:56.292340 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.292901 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.377363 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.392509 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.394531 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.407037 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.407781 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.408494 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.419540 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.421355 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.427299 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.445459 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.448772 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.475021 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:56.476229 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:05:56.476975 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.478698 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.483328 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.491009 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.496021 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.501005 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.507931 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:56.578711 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:56.595367 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:56.606403 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.606937 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.611984 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.622404 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.629397 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.637603 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.670847 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.681233 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.681380 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:56.682826 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:56.696666 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.710634 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.712468 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.715010 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.715498 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.727314 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.732435 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.741228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:56.745054 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:56.748971 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:56.833288 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:56.868465 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.868788 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.878748 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.889464 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.897238 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.907945 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.911546 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.913276 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.922469 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:56.928385 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:56.933506 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:56.934273 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.946483 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:56.948638 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.955081 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:56.963637 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:56.966048 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:56.989151 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:57.000805 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.052980 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.053299 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.054356 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.062060 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.076758 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.077795 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:57.083784 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:57.162149 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:57.163645 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.179287 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.183252 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.185797 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.187152 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.188683 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:57.192106 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:57.198334 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.199422 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.208305 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.209394 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.224812 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.229538 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:57.230809 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:57.239122 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.252238 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.262673 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.267745 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.275105 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.284549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:57.292747 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:57.296087 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.308053 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.309248 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.329425 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.329895 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.355995 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:57.401304 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 14:05:57.407191 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:57.449829 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.451879 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.454751 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.455181 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.456769 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.471560 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.471869 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.481905 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.482002 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.487949 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.496727 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:57.509346 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:57.535943 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:57.540429 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:57.546607 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:57.581537 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.583667 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.587076 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.589453 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.592756 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.624073 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:57.626986 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:57.662977 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:57.675566 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.691025 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.691262 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.691303 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.717223 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.717886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.730298 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.733604 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.739213 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.742704 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:57.747313 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:57.751935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:57.752131 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:57.765475 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:57.776435 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:57.780925 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:57.830014 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:57.831282 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.845075 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.858829 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.863878 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.877123 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.882150 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.899746 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:57.917841 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.920745 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.928680 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.935139 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.962349 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:57.963873 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:57.979703 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:57.980077 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:57.984884 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:57.986038 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.000703 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:58.032548 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 14:05:58.044509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.046357 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.056629 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.057065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.057101 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.059489 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.080086 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.091374 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.102031 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.112143 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.120486 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:58.122382 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:58.126130 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.176185 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.177505 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.178691 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.192066 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.192507 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.195019 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:58.220075 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:58.272167 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.276144 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.277681 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.280221 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.284268 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.287387 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:58.336521 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.356082 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.364454 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.369186 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.369786 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.377143 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.386637 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:58.415421 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:58.468173 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.470408 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.473147 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.479218 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:58.494186 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.496562 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.496902 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.498301 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.503147 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.503644 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.510339 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:58.521744 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.525487 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:58.529503 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:05:58.531316 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.599758 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:58.605742 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.611940 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.616102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:58.622118 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:58.625057 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.630120 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.639658 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:58.648878 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.662874 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:58.665135 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:58.667284 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.668663 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.685273 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.701110 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.703022 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.740702 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.743345 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.747371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.753935 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.754278 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.774571 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.826169 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.828686 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.841996 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.851684 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:58.860488 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.864461 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:58.864662 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:05:58.876400 27760 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:05:58.876507 27760 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:05:58.876930 27760 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:05:58.877970 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:05:58.878109 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:05:58.879312 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:58.879411 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:58.879444 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:05:58.879664 27760 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:05:58.931237 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:05:58.944023 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:05:58.952170 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.964151 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.968096 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:58.969080 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:58.975582 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.975986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:58.979581 29007 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.979676 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:58.980146 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:58.982762 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.008642 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.020448 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.022195 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:59.035199 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:05:59.047463 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:05:59.059412 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:05:59.064443 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:59.064515 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.068043 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.077884 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.081117 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.085703 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.087579 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.099798 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.114934 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.115422 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.131979 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:59.149672 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:59.237042 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.246604 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.262687 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.264286 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.266695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.268796 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.290910 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.295208 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.297183 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.308979 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:59.319499 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:05:59.334859 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.335465 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.335623 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.341915 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.349020 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.349689 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.350033 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.366540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.386787 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.409471 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.412948 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.417551 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.423899 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.436515 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:59.464035 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:59.527762 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.530433 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.541988 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.562278 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.569202 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:59.577418 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:59.582998 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:05:59.607611 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:05:59.629758 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.638175 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.638738 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.639125 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.640439 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.640672 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:59.644536 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:05:59.645612 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.658020 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.684818 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:05:59.685073 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:05:59.687674 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.704425 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.719992 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.738926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.743198 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.743633 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.750159 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:05:59.750849 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:59.752799 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:05:59.757285 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:05:59.771458 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:05:59.774827 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.819128 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.823235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:05:59.828058 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.833900 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.838725 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.849891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.918030 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:05:59.930935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:05:59.932623 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:05:59.932624 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:05:59.935935 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:05:59.947160 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:05:59.984922 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:05:59.989198 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:05:59.989391 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:05:59.998802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.003902 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:00.010936 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:06:00.011575 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.011910 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.013819 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.015652 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.083263 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.094736 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:00.100452 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:00.104375 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.107065 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.109589 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.109589 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.139171 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:00.140656 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:06:00.150930 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.153963 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.168625 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:00.198650 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:06:00.201447 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.207877 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:00.218477 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:06:00.232439 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:06:00.246796 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.251639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.254778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.257938 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.265715 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.270388 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.270815 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.273375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.294481 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.305575 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.306658 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.321961 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:00.329615 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:00.368630 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.369136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.381407 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.381887 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:00.389118 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:00.398939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:00.401422 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:06:00.414388 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.461971 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:00.468703 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:00.469024 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.472401 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.481997 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.491262 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.499154 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.505631 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.532395 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.533744 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.536381 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.539152 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.561527 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:00.600852 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:00.607821 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.622527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.628491 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.633730 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.639659 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.640719 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:00.652360 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 14:06:00.671810 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.678754 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.690611 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.692344 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.698351 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.711937 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.752743 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.761361 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.764776 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.766430 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:00.788007 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.791102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.841682 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.854902 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:00.856992 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 14:06:00.870407 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.872737 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.882159 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.884878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.892441 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.899477 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:00.921610 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:00.925654 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.933194 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.937513 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.953150 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.979265 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:00.983367 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:00.984797 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:00.991086 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:00.992235 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:00.995970 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:01.007459 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:06:01.010608 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:06:01.031455 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:06:01.032124 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:01.061609 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:01.071067 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.076231 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.080250 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.080247 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.080875 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.081236 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:01.082254 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:06:01.098177 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.144899 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.145576 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.147939 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.154358 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:01.157706 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.158960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.163584 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:01.210254 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.224318 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.225271 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.238915 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.241333 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.254756 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.257261 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.260839 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.265267 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.280683 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.280953 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.286394 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.298949 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:01.343547 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.373320 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.374635 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.378696 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.380478 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.392639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.410269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.436043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.439337 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.444455 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.449666 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.454324 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.460419 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:01.504848 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:06:01.505648 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:01.520980 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.525213 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.542356 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.545650 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.552699 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.560217 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.571763 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.574723 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.579311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.587136 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.587929 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.626406 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.648190 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:01.654583 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:01.660274 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.661801 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.663087 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.678639 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.682132 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:01.741114 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.757166 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.768213 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.770433 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:01.783927 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.791354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.800408 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.819176 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.851095 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.859086 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.873471 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.876116 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.890241 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.904672 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.905331 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:01.911584 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:01.927518 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.935989 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.951776 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:01.952859 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.955953 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:01.956529 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.956871 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:01.959856 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:01.961553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:01.981689 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:01.983665 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:01.996065 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:06:02.057253 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.085181 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:02.085204 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:02.088174 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.093596 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.095090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.111462 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.112432 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:02.138208 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:06:02.152806 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:02.157624 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.185842 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.190619 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.198081 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.198171 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.202937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.211939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.213774 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.215895 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.221303 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.232544 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.235066 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.242509 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.264842 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.266170 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.266214 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.267707 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.270932 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:02.288501 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.291846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:02.320544 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.392050 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.401646 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.414986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.421280 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.428459 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:02.430783 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:02.431149 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.431548 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.431787 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.432837 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.435343 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.470729 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.490321 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.528991 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.530086 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.536058 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.541227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.562760 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.566908 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:02.606374 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:02.617801 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.623920 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.626768 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:02.636170 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.636343 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.640647 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.644239 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:02.657675 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:02.680456 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:02.691633 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:02.696079 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.700528 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.700634 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:02.710038 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:02.710767 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.720710 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.763299 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.764753 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:02.773455 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:02.775120 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.776762 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.783315 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.793124 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.837961 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.841523 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.853657 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:02.854718 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.856552 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.867972 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:02.926846 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:02.949834 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:02.953328 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.953513 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:02.954190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:02.961625 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:02.963214 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:02.963440 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:02.988631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:02.997433 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.002220 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.011408 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.019862 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.020936 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:03.030755 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:03.060272 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.060338 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.073197 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.079469 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.080471 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.096554 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:03.106428 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 14:06:03.108672 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 14:06:03.109546 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:03.168257 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:03.187095 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.195925 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.204236 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.225235 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.226644 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.234794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.236771 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.240785 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.249325 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.257331 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.265005 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:03.275501 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:03.289865 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.343405 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.358013 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.362815 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.369810 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:03.374320 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:03.377102 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.377382 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.389369 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.393222 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.396936 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.404377 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.408435 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.420733 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.424600 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.426051 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.437482 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.438773 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.447494 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.459899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.542165 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.543762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.544574 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.544690 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.552563 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:03.560997 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:03.579806 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.606043 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.614246 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.619385 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.632992 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.633505 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:03.645001 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:03.669608 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.670184 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:03.674957 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:03.676100 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:03.677799 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:03.684762 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.692382 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.696166 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:03.706238 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:06:03.722316 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:03.730355 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:03.740870 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:03.742188 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.746858 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.748827 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.750330 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.769131 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:03.777165 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:03.801950 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.837213 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.845731 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.846796 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.852201 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.857228 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.857836 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:03.909778 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:03.909868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:03.910707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:03.915704 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:03.936673 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:03.938764 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.005724 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:06:04.019795 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.026356 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.045151 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.049398 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.054383 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.057371 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.057559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.058043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.085224 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.088563 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.099922 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.101706 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:04.116787 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:04.123584 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:04.174749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.178388 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.179222 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.181560 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:04.183988 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:04.184630 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.187883 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.209400 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.220127 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.221525 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.226090 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:06:04.233201 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.233439 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.240258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:04.273186 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:04.289369 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.289947 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.296010 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.296679 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.302271 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:04.318516 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:04.325907 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.400197 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.407393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.420508 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.420806 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.425642 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.426338 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.428444 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.436681 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.445991 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.449494 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.463933 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:04.473299 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:06:04.474815 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.519513 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.524255 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:04.533751 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:04.534735 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.535218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.537983 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.539232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.554030 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:04.592329 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.602888 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.608434 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.612063 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.614152 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.652415 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.673640 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:04.674867 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.677327 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.681885 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.683210 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.687301 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.689059 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:04.719816 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:06:04.728382 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:04.735344 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:04.745054 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.750077 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.759719 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.771495 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.778242 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.779191 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:06:04.784718 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:04.809190 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.883648 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.886551 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.897506 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.899132 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.901057 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.906379 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:04.909711 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:04.916347 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:04.931624 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:04.937764 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:04.944247 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:04.949700 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:04.950232 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.966549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.970467 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:04.975780 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:04.983951 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.998185 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:04.998993 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.026528 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.061098 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.067523 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.071489 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.077224 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.087713 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.088519 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.120636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:05.123616 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:05.135636 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.136435 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.137868 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.138886 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.142815 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:05.173519 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:05.197510 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:06:05.208106 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:05.209573 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.212749 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.218312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:05.232748 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 14:06:05.237390 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.238953 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.239820 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.281016 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.284400 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.288115 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.297297 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.311357 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.311455 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:05.341895 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:06:05.351380 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:05.390249 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.390560 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.390937 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.414661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.415294 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.415772 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.417462 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.419044 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.426853 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.427114 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.448661 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.452400 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.514904 382 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:05.515036 382 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:05.515507 382 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:05.515897 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:05.516057 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:06:05.517565 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:05.517683 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:05.517735 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:05.517939 382 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:05.536077 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.550274 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.571125 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:05.573051 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:05.580973 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.584573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.590831 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:05.622895 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:05.622895 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 14:06:05.634024 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.635515 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.641374 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.642483 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:05.655778 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:05.659195 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.659749 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.661108 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.665882 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.672048 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.688794 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.692672 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.705981 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.735702 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.749639 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:05.750106 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:05.750519 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.764705 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.770701 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.774093 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:05.834287 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:05.845392 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:05.855659 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:05.859874 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.860152 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.868063 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:05.881063 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.881965 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.900382 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:05.902283 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:05.903190 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:05.907960 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:05.911370 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:05.919696 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:05.983276 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.007158 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.010226 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.011483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.012979 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.014930 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.020541 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:06.041153 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:06:06.042919 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:06.060565 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.063879 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.065423 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.076370 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.087708 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.107151 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:06.118113 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.131798 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.140987 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.144762 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.149051 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.154943 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:06.155264 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.167198 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:06.200906 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:06.218739 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.224360 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.238874 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.241926 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.245584 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.246645 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.250609 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:06.299271 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:06.312597 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:06.315549 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.322528 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.323216 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.342155 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.353570 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.360584 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.364156 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.377708 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.384461 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.399808 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.401733 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.404812 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.405984 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.408294 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.414244 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.426960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:06.437492 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 14:06:06.438495 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.443660 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:06.485673 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.505504 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.507995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.510968 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.511967 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.523240 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.530805 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:06.581101 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:06:06.582389 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:06.638062 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.650238 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.652173 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.657972 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.664620 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:06.665309 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:06.680446 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:06.681622 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.687498 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.688426 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.689975 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.697660 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.700089 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.704663 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.716732 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.717294 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.727329 29138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.727562 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.730516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.746364 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.756461 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:06.758780 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.771091 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.773730 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.798427 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.809002 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:06.868695 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:06.904268 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.907248 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:06.908221 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:06.909034 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.910297 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:06.918200 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:06.966746 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:06.986793 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:06.987843 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:06.991649 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.004417 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.013571 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.014393 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:07.032145 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:07.065703 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.077239 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.077477 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.079882 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.081291 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.085134 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:07.102326 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:07.109102 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:07.169497 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.173420 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.183861 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.204257 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.204582 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.212718 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.218181 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.218755 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.224953 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.226425 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.245523 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.246943 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.252538 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.268682 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.270237 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:07.281718 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:07.283098 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.290647 29404 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.290678 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:07.322496 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:06:07.337998 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:07.343917 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:07.398376 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:07.407730 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:07.423141 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.426867 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.441821 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.448698 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.458241 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.462134 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.462911 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:07.471063 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:07.473668 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.477139 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.479939 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:07.486725 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:07.492745 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.497936 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.520776 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.524135 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.530468 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.542093 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.542989 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.544255 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.626255 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.636689 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.642484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.648435 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.652675 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.653832 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.655035 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.658041 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.659413 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.661016 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.661234 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.670811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:07.675752 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:07.737398 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:07.754180 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.764959 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.766994 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.769261 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.769811 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.786202 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:07.826231 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:07.856354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.871763 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.874614 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:07.881981 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:07.888916 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.889070 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.901659 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:07.903685 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:07.908660 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:07.929286 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:07.941100 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:07.944857 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:07.952729 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:07.955977 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 14:06:07.991638 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:07.997663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.006803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.015089 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.015569 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.020563 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.031075 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.035588 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.039505 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.046553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.056143 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.057924 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.062795 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.171660 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.190232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.194108 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.209192 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.211977 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.215972 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.219466 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.220448 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.226454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.238268 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.238345 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.247247 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.259991 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.299659 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.300474 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.302253 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:08.305963 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:08.313793 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.325171 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:08.346016 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:08.375177 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.376828 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.377487 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.386817 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:08.412897 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:08.421501 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.432709 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.438863 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.444396 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.445930 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.446862 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.455811 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.460660 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:08.462704 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:08.505717 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.528674 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:08.530601 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.535651 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.536473 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.536703 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.537739 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.538143 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.539150 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:08.548028 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:08.550886 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.575227 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.580926 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:08.583220 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.601794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:08.611994 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:08.612751 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.671963 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.678993 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.686995 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.697312 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.698042 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.702564 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.729434 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:08.736539 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.736568 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:08.751946 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.757289 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.763712 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:08.815970 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.818760 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:08.822093 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.823630 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:08.824023 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:08.826664 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:08.827172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.836921 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.849674 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:08.914490 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.918438 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:08.924189 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:08.926232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:08.926484 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:08.932811 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:08.945827 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:08.950060 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:08.978227 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.028080 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.028425 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.038754 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.040813 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.044399 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.059620 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.067891 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.072295 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:09.074875 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 92 similar messages]
I20240502 14:06:09.076016 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.096510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.096640 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.104446 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.104446 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.105834 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.108417 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.117417 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.124261 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.131974 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:09.193002 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 14:06:09.215283 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:09.228085 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:09.246557 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.247623 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.248200 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.255532 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.257557 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:09.302986 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:06:09.303108 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:09.310672 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.314132 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.323487 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.345753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.352099 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:09.361353 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:09.367830 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:09.369370 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:09.370378 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.370591 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.372253 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.377959 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.389940 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.391306 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:09.412762 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:09.440644 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.442405 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.458079 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.459684 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.462313 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.463444 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.494633 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.502063 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.504765 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.508872 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:09.523404 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 14:06:09.530666 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.532603 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:09.573294 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 14:06:09.595682 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:09.599049 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:09.631136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.634358 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.636235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.640542 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.644591 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.670637 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.714135 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.723160 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.724571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.735137 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.736979 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.738561 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.738973 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.739352 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.740772 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.745280 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.757012 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.761276 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.788206 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.797475 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:09.800690 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.806481 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.809954 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:09.828294 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:09.829872 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:09.846546 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:09.850960 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:09.907964 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:06:09.917114 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:09.925961 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:09.931324 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:09.933074 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:09.933497 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:09.933779 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:09.957144 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:09.959954 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:10.007627 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.008882 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:10.019132 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:10.024348 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.025686 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.030858 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.036010 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:10.097790 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:10.111943 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.117720 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.120158 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.121829 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.131103 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.131409 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.136482 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.158195 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.163468 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.165766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.182771 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.183832 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.230417 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.231524 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.235150 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.243206 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.248256 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.272310 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:10.275648 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:10.277038 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:06:10.284761 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:10.313544 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.333447 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.336707 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.348228 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.361780 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.374677 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:10.400379 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:10.401654 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.428886 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.431978 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.434422 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.436553 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.436816 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.440905 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.450796 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.453297 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.466745 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.472460 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.474011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:10.527420 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 14:06:10.556324 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:10.558296 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.558296 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.559440 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.567054 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.571439 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.571645 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.615983 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.619403 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:10.624717 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:10.627848 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.635733 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.638742 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.658583 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.694216 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.696113 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.697548 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.700332 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:10.710534 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:10.725324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.729899 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.791471 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.791920 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.793572 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.795293 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.803344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.806355 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:10.830439 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:10.872566 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:10.873404 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:10.887545 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.888871 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:10.890590 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:10.891244 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:10.892010 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.896406 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:10.932873 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:10.941336 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:10.943945 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:10.957149 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:10.961191 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:10.970578 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:10.991006 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:11.005343 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:06:11.006543 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:11.014431 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.016768 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.018266 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.043571 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.047196 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.068372 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:11.069089 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:06:11.105820 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.111588 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.120676 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.120939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.128651 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.129794 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.189452 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.190100 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.191375 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.195876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.196455 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.196849 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.204711 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.220347 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:11.224900 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:11.229233 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.242444 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.250655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.266206 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:11.311347 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:11.315412 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:11.324900 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.329743 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.330330 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.335561 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.362433 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:11.369297 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:11.370780 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:11.397296 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:11.401712 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:11.401990 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.407948 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.409094 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.417870 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.421252 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.429351 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.465564 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.466807 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.469563 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.470530 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.474102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.488181 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.527720 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.531526 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:11.534477 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:11.540773 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.540993 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.545576 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.555572 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:11.604387 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:11.665297 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.676172 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.689158 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.689713 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.698364 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.708773 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.711477 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.715428 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.716235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.719179 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.726061 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:11.729526 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 14:06:11.730382 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:11.747625 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:11.783473 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:06:11.826566 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 14:06:11.847517 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.852452 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.872223 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.877713 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:11.887568 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.896116 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.902086 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.903818 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.908084 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.932225 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:11.941365 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:11.942102 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:11.955017 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:11.959342 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.962177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.975384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:11.977669 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:11.978274 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:11.984611 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:12.021920 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:12.027762 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:12.050172 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.054924 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.066232 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.067245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.069314 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.076440 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:12.105969 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:12.115160 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:12.131556 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:12.155596 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.157513 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.158294 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.160224 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.169391 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.184168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.195617 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.207645 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.208333 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.227283 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.244169 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.253749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:12.276862 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:12.280829 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.291481 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.297446 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.303822 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.304158 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.304241 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.304981 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.325747 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.336091 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.340258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.355469 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.369781 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:12.399554 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 14:06:12.410863 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:12.432260 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:12.456346 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.459570 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.459688 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.468823 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.482070 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.487290 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.508307 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.513556 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.518030 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.518373 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:12.526631 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:12.529107 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.542009 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:12.547842 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:12.549360 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.555516 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:12.557339 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:12.571148 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.575114 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.576622 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.584614 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:12.649730 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:12.695798 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.706687 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.716478 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.720217 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.734112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.745832 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.759264 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.761776 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.763837 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.769711 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.772145 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.772742 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.806181 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.811923 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.826037 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:12.836601 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:12.838459 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.840204 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.849375 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.855286 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:12.857509 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:12.862632 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.866864 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:12.884691 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.891963 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:12.898720 2801 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:12.898818 2801 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:12.899384 2801 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:12.899873 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:06:12.900027 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:06:12.901257 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:12.901355 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:12.901391 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:12.904518 2801 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:12.959161 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:12.989063 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:12.996093 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:12.998838 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.005219 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.007162 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.016304 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.018170 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.046795 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.057273 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.061959 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.070437 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.072083 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.076640 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.080646 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.082098 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.087606 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:13.095501 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:13.108886 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:13.130085 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:13.136426 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:13.147140 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 14:06:13.206599 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:13.211231 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:13.218621 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:13.249044 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.250329 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.255822 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.269090 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.277734 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.277814 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.289798 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.294786 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.304852 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.307824 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.308995 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.319159 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:13.328357 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:13.333492 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.336359 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.340600 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.346242 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.357043 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:13.372375 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:13.395519 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:13.401705 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:13.431070 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:13.434052 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.451133 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.457310 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.457409 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.469254 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.490068 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.514055 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:13.515225 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:13.521317 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.522428 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.525866 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.527736 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.528379 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.577322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.577610 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.583556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.590978 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.601480 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.627120 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:13.645074 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:13.663480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.663832 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.673740 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.677435 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.678249 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.685534 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.685812 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.689152 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.693805 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.714612 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.721199 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.730463 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:13.772859 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:13.806926 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.813261 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.814235 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.825762 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.826185 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.834825 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:13.866191 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:13.890733 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.893640 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:13.897732 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:13.902374 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:13.904552 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.911559 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.929930 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:13.940016 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:13.941498 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.942328 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:13.943818 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:13.972230 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:13.974814 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:13.991606 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:13.994489 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:13.998993 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:14.010527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.013959 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.022753 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.025242 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.026027 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:14.071848 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 14:06:14.076300 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240502 14:06:14.092931 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:14.097033 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:14.110673 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.110771 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.113461 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.116369 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.128760 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.191864 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.194878 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.217458 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.218988 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.219359 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.236960 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.240605 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.249269 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.254097 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.254674 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.260089 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.269158 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.311447 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.317243 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:14.320770 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:14.328052 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.340970 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.347880 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.374269 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:14.387799 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:14.388671 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:14.388888 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.401654 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.408169 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.409022 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.414223 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:14.432965 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:14.441326 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.484989 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.485375 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.489231 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.498497 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.502890 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.504449 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:14.529739 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:14.547070 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.550650 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.551365 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.553481 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.559660 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:14.560338 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:06:14.584543 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:14.587509 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:14.653439 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:06:14.658234 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:06:14.658501 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.660323 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:14.669096 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:14.670692 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.674394 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.679097 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.679852 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.680940 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.682473 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.692662 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.705559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.710920 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.736712 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:14.758783 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:14.781467 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.781878 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.787601 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:14.794150 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 14:06:14.797031 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:14.797811 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:14.800240 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.804591 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:14.810374 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:14.867998 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.880807 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:14.890982 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:14.893059 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:14.908005 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.911458 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:14.965457 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:14.974458 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:14.977733 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:14.979423 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:14.980724 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:14.989921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.019088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.022789 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.055894 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:15.062199 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:15.069056 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.070905 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.086153 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.089131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.089689 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.093442 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.109253 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.115222 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.116873 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.118248 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.128319 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.132244 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.140568 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.141531 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.148864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.151564 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.156878 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:15.266279 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:15.273036 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:15.274264 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.286311 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.292169 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.297446 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.299968 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.300904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.308459 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.311141 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.328689 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.329408 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.347973 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:15.351528 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:15.371450 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.384469 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.385537 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.387310 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.388176 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.392621 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.399840 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:15.447294 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:15.463297 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.465240 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.468462 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.470989 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.474761 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.491500 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:15.491559 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:06:15.492161 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:15.536091 5845 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:15.536229 5845 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:15.536756 5845 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:15.537206 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:15.537333 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:06:15.541673 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:15.541785 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:15.541833 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:15.542019 5845 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:15.571933 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.580715 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.580847 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:15.582558 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:15.583631 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.584303 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.590898 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.591538 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.591747 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.595387 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.598022 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.619493 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.636924 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:15.674379 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 14:06:15.723495 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.726315 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.733981 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:15.744694 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:15.756076 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:15.762737 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:15.774997 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.778522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.794858 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:15.797574 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:15.805203 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:15.809317 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:15.812728 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.825476 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.834451 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.849100 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.849326 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:15.880887 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:15.885389 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:15.888737 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:15.895280 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:15.905850 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.907348 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:15.917037 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:15.923627 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.925154 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.933128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:15.937980 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:15.940081 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:15.940770 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:15.958962 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.044795 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.054672 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.056864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.062394 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:16.069571 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:16.070459 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.095119 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.100572 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.103454 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.116758 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.123255 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.128674 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.134934 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.137336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.139026 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.152081 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.156793 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.167187 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.168926 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:16.184000 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:06:16.261292 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.271548 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.273010 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.285861 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.303701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.309335 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.314927 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.325796 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.329308 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:16.329313 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:16.329552 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.337913 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.353579 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:16.357714 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 14:06:16.360841 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:16.369832 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.372174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.373039 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.377216 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.377508 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.387293 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.389210 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.396595 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.521041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.534797 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.546280 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.569079 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.575307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.581408 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.588826 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.597119 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.600751 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.614243 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.616040 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.626689 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.632858 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:16.636442 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:16.646111 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.652829 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.675478 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.692646 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.696089 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.698277 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.703583 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.703949 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:16.709501 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.710307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.713327 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.715739 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.721951 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.721964 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.723258 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.723991 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.753481 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.772804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:16.787343 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 14:06:16.805094 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:06:16.841450 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:16.841691 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:16.908037 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.908695 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:16.909572 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.912324 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:16.914841 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.922042 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.930227 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:16.936378 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.944802 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.961311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:16.965065 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.972518 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.974566 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:16.985589 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:16.987903 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:16.991700 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:16.992908 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:16.994537 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.001915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.005555 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:17.024724 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:17.063308 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.069800 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.071094 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.077172 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.087998 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:17.105445 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.108445 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:17.108487 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:17.171368 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.176041 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.181422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.183300 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.192413 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.195222 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.206676 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.215700 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.218995 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.221309 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.225872 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.238639 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.240547 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:17.252593 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:17.319707 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.339807 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.340713 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.358815 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.359160 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.360816 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:17.373456 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:17.374895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.394547 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.394965 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.396786 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:17.414788 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:17.430091 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.436138 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:17.442035 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.467629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.481617 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.483778 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.484831 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.490869 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.506413 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.538051 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.539093 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.541139 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.545058 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.555013 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.557446 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:17.566071 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.621029 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.622067 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.633384 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.650099 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.651803 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.655817 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.672041 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.680408 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.683979 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.686084 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.691026 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.691148 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:17.711956 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.775583 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.780256 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.784909 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:17.802932 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:17.807437 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:17.815960 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:17.817682 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:17.821175 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:17.827908 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:17.831537 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:17.871111 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.876017 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.877779 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:17.881455 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.886807 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.893800 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:17.905419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:17.905982 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:17.907989 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.911725 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.917946 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.925809 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:17.934600 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.937206 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:17.937701 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:17.941035 5845 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:17.941120 5845 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:17.941530 5845 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:17.942385 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:17.942528 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
I20240502 14:06:17.943867 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:17.963929 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:17.964967 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:17.976761 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:17.976881 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:17.976919 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:17.977027 5845 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:17.987388 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:17.991884 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.072082 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.075320 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.092068 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.097491 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.099040 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.100807 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:18.110227 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 14:06:18.204778 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.212779 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.217073 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.218533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.233894 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.236461 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.244309 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.252593 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.255007 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.263481 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.265960 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.282642 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.285332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.286942 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.311069 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.323227 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.325894 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.331540 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.334650 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.337806 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.344506 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.348083 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.361299 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.371357 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:18.373919 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:18.379328 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:18.443735 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.445225 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.450062 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.451591 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:18.461611 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:18.464594 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:18.466375 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.469357 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:18.501318 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.523993 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:18.526115 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.527155 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.533810 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.538519 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.538908 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.555033 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:18.565125 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:06:18.577977 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.600737 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.602416 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.603825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.605908 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.613010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.615310 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.631080 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.636092 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.644073 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:18.648963 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.660580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.669350 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.669760 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.766574 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.771243 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.776373 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:18.785162 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:18.789445 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.801362 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.802567 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.822491 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.826925 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.827250 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.834417 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.848328 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:18.850906 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:18.883224 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:18.947418 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.948978 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:18.950143 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.957427 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.963150 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:18.976441 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:18.982268 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:18.984234 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:18.986606 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:18.988581 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:19.004393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.007086 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.007835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:19.044427 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:19.078691 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:19.085736 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:19.098615 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.110224 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.116058 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.117131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:19.120888 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 95 similar messages]
I20240502 14:06:19.135499 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.171286 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:19.192471 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:19.195927 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.197099 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.206072 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.210191 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.221109 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.238622 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.249917 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.253589 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.270296 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.287153 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.289353 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.295718 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.295955 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.298933 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.300606 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.321990 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:19.325898 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:19.332311 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.349445 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.355998 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.357059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.359452 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:19.368047 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:19.376744 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.402671 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:19.416219 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:19.468864 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.475735 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.476608 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.480700 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.482950 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.493391 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.502400 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.520476 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.529542 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.535763 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.536100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.548911 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:19.552928 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 14:06:19.553081 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:19.559278 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:19.562326 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:19.585361 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.587373 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.587471 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.597592 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.597839 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.599292 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:19.679735 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:06:19.684698 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:19.717867 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:19.719416 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:19.720705 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.723044 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.728142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.751540 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.752065 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.754937 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:19.763760 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:19.770471 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.794574 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:19.794728 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:19.795830 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.801404 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.827101 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.838073 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.851451 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.856196 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:19.880697 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:19.882723 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.884696 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.927284 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:19.944404 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:19.949366 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:19.952018 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:19.953924 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:19.961455 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:19.963200 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:20.029407 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:20.049552 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.055155 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.063053 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.064308 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.066437 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.077936 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.083930 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.090596 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.099840 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.107762 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.119278 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.126125 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.128142 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.135550 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.140478 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.142014 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.144412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:20.156373 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:20.170948 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:20.184183 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:20.210747 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 14:06:20.265197 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.267136 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.271229 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.274040 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.287473 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.292032 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:20.297353 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:20.298511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.315312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:20.326174 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:20.330886 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.332151 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:20.351955 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:20.359630 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.365051 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:20.411607 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:20.412158 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.420034 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.425938 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.426988 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.435097 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.435595 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.440694 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.441457 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.445418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.446765 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:20.465620 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 14:06:20.468251 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.478046 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:20.518642 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:06:20.529297 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:20.560184 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.573719 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.573940 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.580699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.595592 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.622694 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.624326 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.629369 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.647548 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.657040 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.665890 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.673668 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.675940 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.685817 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.693910 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.703423 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.706274 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.713352 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:20.779238 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:20.780054 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.792418 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:20.793565 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.800904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:20.810189 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:20.811064 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.815799 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.818082 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.822924 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.831709 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.857769 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.866935 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.888515 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.938338 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:20.945047 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:20.948444 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:20.953258 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:20.963380 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:20.970944 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:20.984144 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:20.998205 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.001710 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.001760 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.014369 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.040030 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.041252 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.093362 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:21.094053 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.097607 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.099100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.099936 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:21.101761 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:21.119297 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.132447 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.176770 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.186568 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.197818 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.202354 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.202826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.205013 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.210419 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:21.220971 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.223214 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.223572 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.224749 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:21.229794 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 28 similar messages]
I20240502 14:06:21.230402 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:21.230453 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:21.240028 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:21.260493 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:21.275818 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.297082 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:21.297897 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:21.298228 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.306001 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.308360 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.309921 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:21.325421 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.336475 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:21.343614 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.411326 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.415232 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.422726 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.426529 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.426569 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.426904 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.429415 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.445775 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.447351 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.451257 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.456970 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.483652 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.533525 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.536527 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.538324 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.543131 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.545567 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.549245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:21.558229 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:21.584117 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:21.642704 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.644291 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.647320 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.659682 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.662060 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.662457 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.668074 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.671011 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.682278 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.683766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:21.706190 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:21.727147 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.732551 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:21.763794 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:21.774149 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.774245 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.779233 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.798743 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.801293 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.802456 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:21.813457 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 14:06:21.816886 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.832820 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.841356 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.845969 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.850791 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.862604 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.866053 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.921682 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.923779 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:21.929237 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.947750 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:21.948468 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:21.953194 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:21.960220 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:21.960425 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.980654 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:21.982951 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:21.989920 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:21.991173 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:21.999619 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.002419 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:22.022509 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:22.077211 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.077631 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.080351 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.092967 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.094271 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.096655 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:22.125707 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:22.154661 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:22.155552 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.159806 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.161876 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.164242 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.175419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:22.177366 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:22.178278 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:22.184947 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:22.185573 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.208786 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.220650 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.234403 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.236340 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.238421 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.245013 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:22.251005 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:22.335871 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:22.344417 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.349675 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.353513 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.356761 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.361780 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.399262 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.401168 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:22.403019 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:22.405748 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.407555 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.408411 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.428176 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.443787 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.469704 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.473033 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.480604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.490825 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.497148 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.498497 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.539189 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.542513 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.542915 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.544514 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:22.544680 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:22.550938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.552582 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:22.577075 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:22.580083 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.593381 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.599514 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.600019 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.606513 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.606918 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.683286 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.691486 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.694651 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.696074 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.696410 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.696738 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.709816 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.711792 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.732457 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.734246 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.734690 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.749539 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:22.814198 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:22.821578 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.823693 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.828052 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.835075 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:22.842952 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:22.843237 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:22.883383 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:22.889178 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:22.895669 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:22.897002 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:22.897975 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:22.902187 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:22.903419 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 14:06:22.907753 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:22.911571 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:22.913511 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:22.914336 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.002369 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.008086 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.012461 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.017069 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:23.017380 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:23.025571 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.027603 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.067642 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.073513 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.085937 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.086203 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.092108 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.098872 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.160818 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.176503 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.177836 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.183197 8981 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:23.183337 8981 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:23.183696 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.183844 8981 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:06:23.185416 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:23.185688 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:23.185755 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:23.185889 8981 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:23.186367 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:06:23.186488 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:06:23.201218 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.204192 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.253681 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.258759 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.263756 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.266523 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.268168 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.280783 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.284153 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:23.288782 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:23.293162 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.296403 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.296821 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.310957 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.332299 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.353513 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.359771 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.371119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.393410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.398392 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.399726 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:23.411211 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:23.466290 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.470469 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.478392 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.480008 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.484978 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.485488 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:23.525324 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:23.559870 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.561003 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:23.568428 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:23.579078 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:23.584681 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.585340 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.585825 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.587190 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.597893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.602663 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.614080 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.620115 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.641146 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:23.648124 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.659162 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:23.669905 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.672899 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.675990 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.677958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:23.678733 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:23.682984 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.683511 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.711640 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:23.711827 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:23.735025 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.771368 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.772043 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.772922 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:23.789680 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.791230 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.792840 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.793088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:23.824012 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:23.840852 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:23.855473 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.861037 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.871749 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.882687 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:23.884938 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.885726 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:23.897356 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 14:06:23.956367 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.969496 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:23.972503 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:23.978986 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:23.980551 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.991205 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:23.992491 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.002696 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.011816 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.036298 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.037745 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.044071 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.073128 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.093881 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.099093 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.100064 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.105804 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.109468 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.116636 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.124580 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.137909 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.148684 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.152748 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.160164 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.223923 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.228631 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:24.229682 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:24.230767 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.231138 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.235047 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:24.261862 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:24.262583 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.272454 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.275854 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:24.280965 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240502 14:06:24.281234 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:24.287230 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:24.289446 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.295110 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.298300 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.314370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.404055 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.412712 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.422242 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:24.426395 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:24.442402 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:24.447083 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:24.451211 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.460155 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.462396 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.468093 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.478803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.480998 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.486697 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.500113 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.500155 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.509292 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.510442 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.512459 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.515858 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.538938 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:24.585700 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:24.601846 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.603029 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.605175 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.612744 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.617565 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:24.623543 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:06:24.640471 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:24.646485 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.669701 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.676484 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.677871 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.679457 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.681344 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:24.692811 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:24.700680 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:24.702934 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:24.730190 11030 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:24.730315 11030 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
W20240502 14:06:24.730430 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:24.730778 11030 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:24.731389 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
W20240502 14:06:24.731520 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:06:24.731532 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:06:24.732713 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:24.732802 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:24.732832 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:24.732960 11030 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:24.736513 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.741999 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.745839 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.757557 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.757838 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:24.763653 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:24.848558 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:24.868511 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.869864 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:24.879909 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:24.883358 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.892961 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:24.899570 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:24.901028 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.914382 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.931317 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:24.938957 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:24.940546 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.947937 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.951368 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.955610 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:24.963557 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:24.964676 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:24.981150 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.001055 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.003683 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.007782 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.066336 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.067106 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:25.072887 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:25.077147 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.079790 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.097726 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.103181 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.171423 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.176230 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.177234 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.184581 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.207751 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.209043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.211208 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.220485 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.222486 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.223670 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.231142 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.233471 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:25.282266 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
W20240502 14:06:25.288663 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:25.293409 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.295993 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:25.301323 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:25.301606 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.310137 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.311015 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:25.317859 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:25.336263 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.372133 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.373554 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.378543 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.386981 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.402559 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.419799 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:25.437317 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:06:25.449676 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.460832 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.464722 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.464737 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.465356 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.466223 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:25.480996 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 14:06:25.508042 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:25.527808 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.530758 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.531440 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.533221 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:25.540596 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:06:25.546663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:25.549382 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:25.581099 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.582489 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.584657 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.585713 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.585911 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.596285 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.603399 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:25.620186 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:25.665850 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.670802 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.671556 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.678802 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.684897 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.686036 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.754415 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.761585 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.764859 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.765354 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.765412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.768334 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.852778 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:25.862167 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:25.865187 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:25.872725 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:25.878620 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.886947 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.899322 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.914446 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.919019 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:25.921716 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.936555 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.943608 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:25.943825 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.948499 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:25.966158 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:25.981065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.983206 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:25.986320 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:25.990856 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.008232 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:26.048394 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:26.073819 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:26.084503 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:26.093271 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.102092 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:26.110136 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:26.114179 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.122401 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.131318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:26.177497 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:26.184181 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:26.207846 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.211270 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.220518 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.223608 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.224825 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.225323 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.233686 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.236626 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.240517 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.247479 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.261967 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.262744 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:26.272804 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 14:06:26.273716 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:26.274327 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:06:26.281256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.284586 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.297376 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:26.302821 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:06:26.303844 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:26.318874 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.324354 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:26.334288 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:26.334779 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.348500 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.351007 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.360518 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.373312 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:26.378019 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:26.382246 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:26.390134 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:26.392370 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:26.467599 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.470953 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.486894 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.491235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.499881 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.524009 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.535663 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.537868 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.549451 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.554541 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.558966 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.563789 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.572007 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.586300 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:26.589388 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:26.595111 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.606573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.606639 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.619446 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.692970 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.693742 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.696632 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.698493 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.702900 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.713685 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.763039 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.763715 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:26.769623 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.776436 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.793913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:26.795531 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.808210 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:26.809489 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.812669 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.822328 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.824334 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:26.863489 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:26.875022 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:06:26.920656 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:26.954519 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:26.956733 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:26.974287 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:26.981648 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 14:06:26.982831 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:26.988759 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:26.997393 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.001421 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.005261 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.005471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.018455 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.031266 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.045020 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.048939 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:27.061849 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:27.077633 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:27.081300 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:27.088848 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.110916 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.113224 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.113610 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.128481 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:27.129491 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:27.134174 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.140969 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.143707 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.147692 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.150071 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:27.159588 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:27.160053 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.248034 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.253599 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.254460 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.260040 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.275321 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.276412 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.338508 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.340480 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:27.351838 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:27.353694 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:27.354328 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:27.375835 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:27.384222 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:27.401633 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.404667 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.408671 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.413239 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.415829 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.415879 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.420486 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.431143 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.444829 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.448563 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:27.449595 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:27.480674 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.483765 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.488569 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.515710 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.535085 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:27.536731 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:27.539760 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:27.547755 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:27.549103 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.549686 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.560467 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.563366 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:27.577291 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:27.577674 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.622629 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.628383 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.633843 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.637307 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.647541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.648697 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.701157 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.704999 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.711280 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:27.722074 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:27.735673 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.737198 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.748373 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.779839 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.785240 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.796028 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:27.796912 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.797936 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.812141 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.876639 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:27.878764 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:27.883877 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.889235 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.902218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.907686 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.908182 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:27.916234 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:06:27.960552 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 14:06:27.963150 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:27.965318 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:27.973547 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:27.973608 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:27.984095 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:27.995811 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:27.998808 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.054544 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.066357 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.066586 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.067692 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.071816 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.089774 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:28.095579 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:28.095989 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.100199 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.105451 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.108919 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.118011 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.119702 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:28.157718 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:28.210990 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.220712 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.224496 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.231691 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.259277 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.261185 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.277618 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.280148 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.282006 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.301307 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.302348 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.303915 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.361567 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.363070 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.365444 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:28.376716 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:28.377266 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.386420 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.399439 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.400465 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.410693 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.413519 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.422978 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.431875 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.434376 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.495530 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.499379 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.499748 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.508423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:28.517078 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:28.518198 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.531385 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.563833 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.564399 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:28.573166 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:28.574354 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.587886 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.591629 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.593701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:28.601667 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:28.606702 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.609707 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.612634 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.627908 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.629348 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.649169 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:28.678612 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:28.747361 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.751484 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.756493 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.771946 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.774400 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.774621 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.790689 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:28.796350 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.797892 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:28.801636 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:28.803275 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:28.804850 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:28.835487 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:28.841244 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:28.842562 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:28.850040 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:28.853506 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.858578 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:28.862246 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:28.873692 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:28.877228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:28.878345 29226 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:28.884534 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 14:06:28.901536 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:28.905345 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:06:28.967355 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:29.019416 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.029742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.032534 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.053148 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.055269 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:29.055872 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:29.062747 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.064244 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.079174 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.091240 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.112780 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.122942 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.128491 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.132392 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.136214 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.137079 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.165730 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.168406 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.169751 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.181330 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.184887 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.193120 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.196698 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.200238 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.203259 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:29.212850 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:29.222481 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.231137 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.246752 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.248922 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.248952 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.254452 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:29.311465 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240502 14:06:29.337781 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:06:29.379462 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.386915 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.389606 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.392284 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.395449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.402622 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.411113 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.419761 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:29.422343 29094 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:29.440716 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.454052 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.474270 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.475657 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.477399 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.478483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.485951 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.506841 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.509883 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:29.512194 28961 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:29.512832 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:29.529842 28961 consensus_peers.cc:485] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:06:29.534796 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:29.557130 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.557634 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.560030 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.563867 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.580307 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.603387 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.651106 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.654001 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.658644 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.665706 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.669849 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.674916 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.680367 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.683934 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:29.691394 29094 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:29.692766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.693408 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.711584 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.715123 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:29.726794 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:29.770139 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:29.784289 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 14:06:29.790066 30849 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 14:06:29.823716 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:06:29.836602 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.839113 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.847041 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.861152 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.871734 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.872215 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.907714 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.907964 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.916111 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:29.929140 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:29.932529 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.937976 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:29.944587 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.946470 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:29.947449 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.963949 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:29.970669 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:29.988708 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:30.023844 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:30.075060 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:30.083604 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:30.087941 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.088181 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:30.097170 31116 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:30.103142 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.115017 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.115891 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.147441 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.153527 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.155344 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.155942 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:30.170354 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:30.174253 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.183444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:30.183899 29226 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:30.185743 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.215646 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.216424 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.221597 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.226861 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.233954 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.240383 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.244786 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.252475 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.269119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.272413 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.279886 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:30.292851 29094 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:30.359611 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.360711 14110 raft_consensus.cc:998] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.157s)
I20240502 14:06:30.360837 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.361073 14110 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:30.362087 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.363137 30915 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.364058 13482 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.370286 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.380996 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.381927 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.392084 14110 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.392534 30915 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.395048 28884 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
W20240502 14:06:30.439038 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:30.441525 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.443298 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.445183 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.450281 29028 consensus_queue.cc:237] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
W20240502 14:06:30.450487 31116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 14:06:30.452867 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.454854 12022 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:30.454957 12022 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:30.455535 12022 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
W20240502 14:06:30.458098 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:30.458185 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:30.458216 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:30.458585 30915 raft_consensus.cc:1261] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:30.458832 12022 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:30.460642 28961 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:30.463793 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:30.464651 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.464805 14110 consensus_queue.cc:1035] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:30.465436 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:06:30.465553 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:06:30.466320 13482 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:30.466631 30915 raft_consensus.cc:2932] T 5a91fd318f0844c1961f16823a7ccc88 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:30.468485 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5a91fd318f0844c1961f16823a7ccc88 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.470029 28880 catalog_manager.cc:5484] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:30.475404 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.479027 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.482954 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:30.490865 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a91fd318f0844c1961f16823a7ccc88: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:30.492441 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.503374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.506721 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.514043 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.515570 12022 raft_consensus.cc:998] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.360s)
I20240502 14:06:30.515897 12022 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:30.520094 30915 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.521004 12022 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.530632 14302 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.533555 30915 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.534032 28884 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.545390 29292 consensus_queue.cc:237] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
W20240502 14:06:30.547119 29226 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): Couldn't send request to peer bd40abfe1fbd48529f9ebbcc4d3abf86. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:30.550164 28961 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:30.552311 30915 raft_consensus.cc:1261] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:30.552824 12022 consensus_queue.cc:1035] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:30.554487 12022 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:30.554749 30915 raft_consensus.cc:2932] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:30.557420 28884 catalog_manager.cc:5484] T a75497cf22014c0cb9d079b448284b3d P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 8 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:06:30.558127 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:06:30.559849 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a75497cf22014c0cb9d079b448284b3d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.585609 14322 raft_consensus.cc:998] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.246s)
I20240502 14:06:30.585938 14322 consensus_queue.cc:237] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:30.587379 29028 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:06:30.587886 12022 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:30.590087 14302 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.590479 29028 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.593331 28884 catalog_manager.cc:5484] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.597121 29292 consensus_queue.cc:237] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:30.598434 29028 raft_consensus.cc:1261] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:30.598855 14302 consensus_queue.cc:1035] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:06:30.600606 29226 consensus_peers.cc:485] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): Couldn't send request to peer bd40abfe1fbd48529f9ebbcc4d3abf86. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0669ab13586a48ff9382fe1b80f47553. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.601260 12022 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:30.603623 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0669ab13586a48ff9382fe1b80f47553 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.604118 29028 raft_consensus.cc:2932] T 0669ab13586a48ff9382fe1b80f47553 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:06:30.604378 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:30.606197 28880 catalog_manager.cc:5484] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 6 to 7, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:06:30.611203 29358 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:30.615795 14288 raft_consensus.cc:998] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.226s)
I20240502 14:06:30.616217 14288 consensus_queue.cc:237] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:30.617650 29426 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:30.618042 31178 raft_consensus.cc:1261] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:30.618436 13482 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:06:30.618618 14110 consensus_queue.cc:1035] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:06:30.620966 14110 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.622831 31178 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
W20240502 14:06:30.626278 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:30.626442 28880 catalog_manager.cc:5484] T d9e0261898a445949f3cf21d2efb599f P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.636857 29426 raft_consensus.cc:2932] T d9e0261898a445949f3cf21d2efb599f P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.637318 14288 raft_consensus.cc:998] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.061s)
I20240502 14:06:30.637765 14288 consensus_queue.cc:237] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } }
I20240502 14:06:30.639170 29292 raft_consensus.cc:1261] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:30.639590 13482 consensus_queue.cc:1035] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:06:30.641064 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.641600 13482 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:06:30.643184 29292 raft_consensus.cc:2932] T db68071f838a4e209f3db69bea6e54f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:06:30.644482 28880 catalog_manager.cc:5484] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 6 to 7, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } }
I20240502 14:06:30.656123 14302 raft_consensus.cc:998] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.202s)
I20240502 14:06:30.656591 14302 consensus_queue.cc:237] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:30.658804 29426 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.658972 31178 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.659217 30915 raft_consensus.cc:1261] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.659602 12022 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.659780 14323 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.660132 14323 consensus_queue.cc:1035] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.667518 12022 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.670107 30915 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.670269 11600 raft_consensus.cc:998] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.264s)
I20240502 14:06:30.670612 11600 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:30.671994 29028 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:06:30.672647 29426 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.673216 28882 catalog_manager.cc:5484] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.679512 11600 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:30.680586 31178 raft_consensus.cc:2932] T ff65ce0568514e89894ec845fbda31f3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.685637 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.688253 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.695708 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.697856 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.698421 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.714102 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.727895 11600 raft_consensus.cc:998] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.291s)
I20240502 14:06:30.727895 12022 raft_consensus.cc:998] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.322s)
I20240502 14:06:30.728205 12022 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:30.728585 11600 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:30.730139 29426 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:06:30.730544 14323 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:30.732095 31178 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:30.732718 14451 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.757853 12819 raft_consensus.cc:998] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.376s)
I20240502 14:06:30.758222 12819 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:30.760038 29159 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:06:30.760949 29426 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer 12348ca540074b32912fdd6333484e0b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:06:30.761421 12819 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.761721 12819 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:30.764995 12819 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.767805 29159 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.768622 28875 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.773921 29426 raft_consensus.cc:2932] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.776652 14451 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.779129 28875 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.779841 29028 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:30.793172 14482 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.795360 31178 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.795866 28878 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:30.797120 14323 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.799916 29426 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.802544 28878 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 5 to 6, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.804052 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.806236 29292 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:30.809399 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.809425 29426 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:06:30.809823 29225 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0176fb1a25a94c1c9d8cf0dc63577e91. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.809937 14302 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:06:30.811156 29226 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 14:06:30.815567 14302 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.817890 29426 raft_consensus.cc:2932] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.818343 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.818979 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.819176 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.824579 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.830288 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.833453 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.838486 28878 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 6 to 7, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:30.841519 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:30.844415 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.844444 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.851675 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.852090 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.852385 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.853976 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:30.868187 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:30.869570 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:30.886579 29358 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:30.896121 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.898816 29159 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:30.898816 29160 consensus_queue.cc:237] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:30.903859 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:30.905041 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:30.907002 29028 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:30.908305 14482 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:06:30.911496 14482 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.913460 29028 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.913892 28878 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 6 to 7, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.916332 31178 raft_consensus.cc:1261] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:30.916752 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.916739 14451 consensus_queue.cc:1035] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:30.931809 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:30.933426 29095 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): Couldn't send request to peer cd692a8958c241fca3aec5cca3a0ded5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dda4193aaaf459fb5049ff49541229a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.934715 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1dda4193aaaf459fb5049ff49541229a with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.935191 14451 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.938028 28875 catalog_manager.cc:5484] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.938787 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:30.949401 29095 consensus_peers.cc:485] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): Couldn't send request to peer cd692a8958c241fca3aec5cca3a0ded5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86cfc7d5709f414aac098ae3c05a48a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:30.951102 31178 raft_consensus.cc:2932] T 86cfc7d5709f414aac098ae3c05a48a3 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:30.952545 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 86cfc7d5709f414aac098ae3c05a48a3 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:30.953088 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:30.959389 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.010283 14569 ts_tablet_manager.cc:902] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:31.010957 14569 tablet_copy_client.cc:308] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:06:31.015811 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 5a91fd318f0844c1961f16823a7ccc88 from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:56987)
I20240502 14:06:31.015930 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 5a91fd318f0844c1961f16823a7ccc88 from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:56987: session id = edce308627b14fa6968d90ec77d51aec-5a91fd318f0844c1961f16823a7ccc88
I20240502 14:06:31.017459 29048 tablet_copy_source_session.cc:205] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.019452 14569 ts_tablet_manager.cc:1875] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:31.025154 14569 ts_tablet_manager.cc:1888] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:06:31.025336 14569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a91fd318f0844c1961f16823a7ccc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.026471 14569 tablet_copy_client.cc:783] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.032011 14569 tablet_copy_client.cc:647] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.033443 14569 tablet_copy_client.cc:523] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.035265 14569 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:06:31.042153 14594 ts_tablet_manager.cc:902] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:06:31.044678 14569 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.045030 14569 tablet_bootstrap.cc:492] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:31.045300 14569 ts_tablet_manager.cc:1367] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240502 14:06:31.045783 14569 raft_consensus.cc:348] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:31.045974 14569 raft_consensus.cc:729] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.046139 14569 consensus_queue.cc:260] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:31.046470 14569 ts_tablet_manager.cc:1398] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:06:31.047048 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-5a91fd318f0844c1961f16823a7ccc88 received from {username='slave'} at 127.16.217.4:56987
I20240502 14:06:31.047122 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session edce308627b14fa6968d90ec77d51aec-5a91fd318f0844c1961f16823a7ccc88 on tablet 5a91fd318f0844c1961f16823a7ccc88 with peer edce308627b14fa6968d90ec77d51aec
I20240502 14:06:31.049155 14594 tablet_copy_client.cc:308] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:06:31.049785 14595 ts_tablet_manager.cc:902] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:06:31.050307 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 0669ab13586a48ff9382fe1b80f47553 from peer bd40abfe1fbd48529f9ebbcc4d3abf86 ({username='slave'} at 127.16.217.2:59669)
I20240502 14:06:31.050410 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 0669ab13586a48ff9382fe1b80f47553 from peer bd40abfe1fbd48529f9ebbcc4d3abf86 at {username='slave'} at 127.16.217.2:59669: session id = bd40abfe1fbd48529f9ebbcc4d3abf86-0669ab13586a48ff9382fe1b80f47553
I20240502 14:06:31.051925 29313 tablet_copy_source_session.cc:205] T 0669ab13586a48ff9382fe1b80f47553 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.052740 14595 tablet_copy_client.cc:308] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:06:31.056459 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet a75497cf22014c0cb9d079b448284b3d from peer bd40abfe1fbd48529f9ebbcc4d3abf86 ({username='slave'} at 127.16.217.2:59669)
I20240502 14:06:31.056560 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet a75497cf22014c0cb9d079b448284b3d from peer bd40abfe1fbd48529f9ebbcc4d3abf86 at {username='slave'} at 127.16.217.2:59669: session id = bd40abfe1fbd48529f9ebbcc4d3abf86-a75497cf22014c0cb9d079b448284b3d
I20240502 14:06:31.057490 29313 tablet_copy_source_session.cc:205] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.058471 14595 ts_tablet_manager.cc:1875] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:31.058660 14594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0669ab13586a48ff9382fe1b80f47553. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:06:31.060155 29358 consensus_peers.cc:462] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:31.061131 14595 ts_tablet_manager.cc:1888] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:06:31.061383 14595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a75497cf22014c0cb9d079b448284b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.062760 14595 tablet_copy_client.cc:783] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.072067 14594 tablet_copy_client.cc:783] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.080777 14594 tablet_copy_client.cc:647] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.088580 14594 tablet_copy_client.cc:523] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.090186 14594 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:06:31.110350 14594 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.110901 14594 tablet_bootstrap.cc:492] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap complete.
I20240502 14:06:31.111150 14594 ts_tablet_manager.cc:1367] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20240502 14:06:31.111711 14594 raft_consensus.cc:348] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:31.111891 14594 raft_consensus.cc:729] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.112097 14594 consensus_queue.cc:260] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:31.113118 14594 ts_tablet_manager.cc:1398] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:06:31.113687 14595 tablet_copy_client.cc:647] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.113663 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-0669ab13586a48ff9382fe1b80f47553 received from {username='slave'} at 127.16.217.2:59669
I20240502 14:06:31.113756 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-0669ab13586a48ff9382fe1b80f47553 on tablet 0669ab13586a48ff9382fe1b80f47553 with peer bd40abfe1fbd48529f9ebbcc4d3abf86
I20240502 14:06:31.116233 14595 tablet_copy_client.cc:523] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.117671 14595 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:06:31.124245 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.129623 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.144701 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.156993 12668 raft_consensus.cc:998] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.516s)
I20240502 14:06:31.157302 12668 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:31.158170 14595 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.158425 14595 tablet_bootstrap.cc:492] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap complete.
I20240502 14:06:31.158457 31178 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:31.158623 14595 ts_tablet_manager.cc:1367] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.004s
I20240502 14:06:31.159032 14676 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:31.159042 14595 raft_consensus.cc:348] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:31.159147 14595 raft_consensus.cc:729] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.159659 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.159613 14595 consensus_queue.cc:260] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:31.160197 14595 ts_tablet_manager.cc:1398] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:06:31.160785 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-a75497cf22014c0cb9d079b448284b3d received from {username='slave'} at 127.16.217.2:59669
I20240502 14:06:31.160851 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-a75497cf22014c0cb9d079b448284b3d on tablet a75497cf22014c0cb9d079b448284b3d with peer bd40abfe1fbd48529f9ebbcc4d3abf86
I20240502 14:06:31.164741 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.165292 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.175076 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88): 4840 bytes on disk
I20240502 14:06:31.175484 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(5a91fd318f0844c1961f16823a7ccc88) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.182766 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.192953 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.197289 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.197295 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:31.206549 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:31.208302 14694 ts_tablet_manager.cc:902] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:06:31.208794 14694 tablet_copy_client.cc:308] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:06:31.212081 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 0176fb1a25a94c1c9d8cf0dc63577e91 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:59889)
I20240502 14:06:31.212196 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 0176fb1a25a94c1c9d8cf0dc63577e91 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:59889: session id = 12348ca540074b32912fdd6333484e0b-0176fb1a25a94c1c9d8cf0dc63577e91
I20240502 14:06:31.213208 29313 tablet_copy_source_session.cc:205] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.220510 14694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0176fb1a25a94c1c9d8cf0dc63577e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.223873 14694 tablet_copy_client.cc:783] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.224617 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:31.226727 30849 consensus_peers.cc:485] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:31.227705 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.233929 14676 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.235842 31178 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.237119 28875 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 7 to 8, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.244910 29426 consensus_queue.cc:237] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:31.246402 31178 raft_consensus.cc:1261] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:31.246843 14676 consensus_queue.cc:1035] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:06:31.248608 29359 consensus_peers.cc:485] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec -> Peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): Couldn't send request to peer a7221b90d0aa4770a8ce5522103a96f4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.249907 14676 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:31.252267 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f05a1a0bf1314d098f2bf65df58222e2 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:31.252573 31181 raft_consensus.cc:2932] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
W20240502 14:06:31.256816 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:31.256917 28878 catalog_manager.cc:5484] T f05a1a0bf1314d098f2bf65df58222e2 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 8 to 9, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:31.265314 14694 tablet_copy_client.cc:647] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.271759 14694 tablet_copy_client.cc:523] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.273510 14694 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:06:31.281445 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.287349 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.292033 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.293169 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.295598 14694 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.295897 14694 tablet_bootstrap.cc:492] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:06:31.296073 14694 ts_tablet_manager.cc:1367] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.000s
I20240502 14:06:31.296469 14694 raft_consensus.cc:348] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:31.296584 14694 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.296711 14694 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:31.297039 14694 ts_tablet_manager.cc:1398] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:06:31.299588 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91): 4586 bytes on disk
I20240502 14:06:31.299989 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(0176fb1a25a94c1c9d8cf0dc63577e91) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.300397 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-0176fb1a25a94c1c9d8cf0dc63577e91 received from {username='slave'} at 127.16.217.6:59889
I20240502 14:06:31.300472 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session 12348ca540074b32912fdd6333484e0b-0176fb1a25a94c1c9d8cf0dc63577e91 on tablet 0176fb1a25a94c1c9d8cf0dc63577e91 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:06:31.305436 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.306116 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.313038 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 14:06:31.314014 29094 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:06:31.318503 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.321766 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.345989 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.352281 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.362615 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d): 4925 bytes on disk
I20240502 14:06:31.364353 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(a75497cf22014c0cb9d079b448284b3d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.365103 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553): 4586 bytes on disk
I20240502 14:06:31.368604 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(0669ab13586a48ff9382fe1b80f47553) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.386099 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:31.390780 28961 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:31.400478 14774 ts_tablet_manager.cc:902] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:06:31.401165 14774 tablet_copy_client.cc:308] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:06:31.401645 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 1dda4193aaaf459fb5049ff49541229a from peer cd692a8958c241fca3aec5cca3a0ded5 ({username='slave'} at 127.16.217.3:53611)
I20240502 14:06:31.401739 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 1dda4193aaaf459fb5049ff49541229a from peer cd692a8958c241fca3aec5cca3a0ded5 at {username='slave'} at 127.16.217.3:53611: session id = cd692a8958c241fca3aec5cca3a0ded5-1dda4193aaaf459fb5049ff49541229a
I20240502 14:06:31.402886 29181 tablet_copy_source_session.cc:205] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.404170 14774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.407107 14774 tablet_copy_client.cc:783] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.412529 14774 tablet_copy_client.cc:647] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.415253 14774 tablet_copy_client.cc:523] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.417033 14774 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:06:31.429854 14774 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.430176 14774 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap complete.
I20240502 14:06:31.430393 14774 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240502 14:06:31.430847 14774 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:31.430964 14774 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.431092 14774 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:31.431752 14774 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 14:06:31.432205 14800 ts_tablet_manager.cc:902] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:06:31.432458 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session cd692a8958c241fca3aec5cca3a0ded5-1dda4193aaaf459fb5049ff49541229a received from {username='slave'} at 127.16.217.3:53611
I20240502 14:06:31.432540 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session cd692a8958c241fca3aec5cca3a0ded5-1dda4193aaaf459fb5049ff49541229a on tablet 1dda4193aaaf459fb5049ff49541229a with peer cd692a8958c241fca3aec5cca3a0ded5
I20240502 14:06:31.432720 14800 tablet_copy_client.cc:308] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:06:31.433359 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer cd692a8958c241fca3aec5cca3a0ded5 ({username='slave'} at 127.16.217.3:53611)
I20240502 14:06:31.433459 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 86cfc7d5709f414aac098ae3c05a48a3 from peer cd692a8958c241fca3aec5cca3a0ded5 at {username='slave'} at 127.16.217.3:53611: session id = cd692a8958c241fca3aec5cca3a0ded5-86cfc7d5709f414aac098ae3c05a48a3
I20240502 14:06:31.434525 29181 tablet_copy_source_session.cc:205] T 86cfc7d5709f414aac098ae3c05a48a3 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.435950 14800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86cfc7d5709f414aac098ae3c05a48a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.438225 14800 tablet_copy_client.cc:783] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:31.442986 14800 tablet_copy_client.cc:647] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.443240 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.444850 14800 tablet_copy_client.cc:523] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.446182 14800 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
I20240502 14:06:31.450801 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.454833 14800 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.455127 14800 tablet_bootstrap.cc:492] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap complete.
I20240502 14:06:31.455257 14800 ts_tablet_manager.cc:1367] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240502 14:06:31.455600 14800 raft_consensus.cc:348] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:31.455686 14800 raft_consensus.cc:729] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.455794 14800 consensus_queue.cc:260] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:31.456014 14800 ts_tablet_manager.cc:1398] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:06:31.456442 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session cd692a8958c241fca3aec5cca3a0ded5-86cfc7d5709f414aac098ae3c05a48a3 received from {username='slave'} at 127.16.217.3:53611
I20240502 14:06:31.456506 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session cd692a8958c241fca3aec5cca3a0ded5-86cfc7d5709f414aac098ae3c05a48a3 on tablet 86cfc7d5709f414aac098ae3c05a48a3 with peer cd692a8958c241fca3aec5cca3a0ded5
I20240502 14:06:31.460757 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:31.468899 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:31.470857 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.472327 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.475565 29426 raft_consensus.cc:1203] T 5a91fd318f0844c1961f16823a7ccc88 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:31.485297 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a): 4840 bytes on disk
I20240502 14:06:31.485698 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.486392 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3): 4840 bytes on disk
I20240502 14:06:31.486708 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(86cfc7d5709f414aac098ae3c05a48a3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.492318 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:31.497010 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a476d1664de7456ab21ddbf2e12b4f0d: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:31.500676 29159 raft_consensus.cc:1203] T a75497cf22014c0cb9d079b448284b3d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:31.507359 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.507627 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.511616 29159 raft_consensus.cc:1203] T 0669ab13586a48ff9382fe1b80f47553 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:06:31.514863 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.514995 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.539044 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.540797 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.569034 12668 raft_consensus.cc:998] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.056s)
I20240502 14:06:31.569393 12668 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:31.570945 31181 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:06:31.571339 14676 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:31.572983 12668 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.573312 31181 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.575287 28878 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:31.578768 29426 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:31.580030 31181 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240502 14:06:31.580205 29359 consensus_peers.cc:485] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec -> Peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835): Couldn't send request to peer a7221b90d0aa4770a8ce5522103a96f4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15513e220e144883b216c4f3ac8ade0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.580420 14676 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:06:31.582224 12668 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:31.583725 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 15513e220e144883b216c4f3ac8ade0b with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:31.585666 31181 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:31.588198 28878 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 9 to 10, VOTER a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1) added. New cstate: current_term: 2 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
W20240502 14:06:31.615017 29094 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 14:06:31.630424 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.631947 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.635396 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.635622 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.645692 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.673357 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:31.674162 30849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:06:31.674722 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:31.690140 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.695690 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.699504 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.711946 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.716642 14961 ts_tablet_manager.cc:902] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:06:31.717200 14961 tablet_copy_client.cc:308] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:06:31.720464 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.720634 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet f05a1a0bf1314d098f2bf65df58222e2 from peer a7221b90d0aa4770a8ce5522103a96f4 ({username='slave'} at 127.16.217.1:46147)
I20240502 14:06:31.720702 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet f05a1a0bf1314d098f2bf65df58222e2 from peer a7221b90d0aa4770a8ce5522103a96f4 at {username='slave'} at 127.16.217.1:46147: session id = a7221b90d0aa4770a8ce5522103a96f4-f05a1a0bf1314d098f2bf65df58222e2
I20240502 14:06:31.721894 29448 tablet_copy_source_session.cc:205] T f05a1a0bf1314d098f2bf65df58222e2 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:31.723066 14961 ts_tablet_manager.cc:1875] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:31.725996 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.738737 14961 ts_tablet_manager.cc:1888] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:06:31.738976 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05a1a0bf1314d098f2bf65df58222e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:31.740206 14961 tablet_copy_client.cc:783] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Starting download of 5 data blocks...
W20240502 14:06:31.748486 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:06:31.758613 14961 tablet_copy_client.cc:647] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:31.759774 14961 tablet_copy_client.cc:523] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:31.761767 14961 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:06:31.777580 14961 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:31.777930 14961 tablet_bootstrap.cc:492] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap complete.
I20240502 14:06:31.778182 14961 ts_tablet_manager.cc:1367] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.002s
I20240502 14:06:31.778743 14961 raft_consensus.cc:348] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:31.778941 14961 raft_consensus.cc:729] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:06:31.779389 14961 consensus_queue.cc:260] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:31.780579 14961 ts_tablet_manager.cc:1398] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:06:31.781502 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session a7221b90d0aa4770a8ce5522103a96f4-f05a1a0bf1314d098f2bf65df58222e2 received from {username='slave'} at 127.16.217.1:46147
I20240502 14:06:31.781577 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session a7221b90d0aa4770a8ce5522103a96f4-f05a1a0bf1314d098f2bf65df58222e2 on tablet f05a1a0bf1314d098f2bf65df58222e2 with peer a7221b90d0aa4770a8ce5522103a96f4
I20240502 14:06:31.790895 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.795331 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.795843 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.804874 30915 raft_consensus.cc:1203] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:06:31.806653 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.826884 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.835855 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.842559 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2): 4925 bytes on disk
I20240502 14:06:31.843020 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(f05a1a0bf1314d098f2bf65df58222e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:06:31.883863 14110 raft_consensus.cc:998] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.222s)
I20240502 14:06:31.884202 14110 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:31.885027 29292 raft_consensus.cc:1203] T 86cfc7d5709f414aac098ae3c05a48a3 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:31.885489 30915 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:31.885874 14288 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:31.887816 14288 raft_consensus.cc:998] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.049s)
I20240502 14:06:31.888056 14288 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:31.889012 30916 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:31.889355 14288 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:31.895408 14110 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:31.897626 14288 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:31.898483 28879 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:31.898901 30915 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:31.899240 30916 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:31.907445 29028 consensus_queue.cc:237] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:31.907651 28877 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
W20240502 14:06:31.909888 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.910197 30916 raft_consensus.cc:1261] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:31.912379 14288 consensus_queue.cc:1035] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:31.913015 29028 consensus_queue.cc:237] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:31.914518 30916 raft_consensus.cc:1261] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:31.914476 14110 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:06:31.917766 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:31.917932 14288 consensus_queue.cc:1035] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:31.919653 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 81b2d5d0f4044106bed079c3faa72447 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:31.920078 14288 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:31.920313 29292 raft_consensus.cc:1203] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:06:31.921746 30916 raft_consensus.cc:2932] T 81b2d5d0f4044106bed079c3faa72447 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:31.922077 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c88f85f143d845a1b7066244c606f8ba with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:31.922472 30915 raft_consensus.cc:2932] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:31.923131 28876 catalog_manager.cc:5484] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:31.924973 28884 catalog_manager.cc:5484] T c88f85f143d845a1b7066244c606f8ba P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 8 to 9, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:31.937492 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.938962 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:31.949532 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.965515 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.970819 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:31.975711 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.976742 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:31.977020 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:31.980474 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.987709 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:31.993468 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:31.995792 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.004777 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.006850 14961 ts_tablet_manager.cc:902] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:06:32.007347 14961 tablet_copy_client.cc:308] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:06:32.007809 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet 15513e220e144883b216c4f3ac8ade0b from peer a7221b90d0aa4770a8ce5522103a96f4 ({username='slave'} at 127.16.217.1:46147)
I20240502 14:06:32.007896 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet 15513e220e144883b216c4f3ac8ade0b from peer a7221b90d0aa4770a8ce5522103a96f4 at {username='slave'} at 127.16.217.1:46147: session id = a7221b90d0aa4770a8ce5522103a96f4-15513e220e144883b216c4f3ac8ade0b
I20240502 14:06:32.008944 29448 tablet_copy_source_session.cc:205] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.009959 14961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.012312 14961 tablet_copy_client.cc:783] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.025353 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.026538 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.037881 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.039897 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.044205 14961 tablet_copy_client.cc:647] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.049845 14961 tablet_copy_client.cc:523] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.051324 14961 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap starting.
I20240502 14:06:32.055984 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.102998 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.114410 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.115094 14049 raft_consensus.cc:998] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Attempting to remove follower 33ebf308f6f24e6e8768ef5f14ddc785 from the Raft config. Reason: Leader has been unable to successfully communicate with peer 33ebf308f6f24e6e8768ef5f14ddc785 for more than 300 seconds (300.454s)
I20240502 14:06:32.115419 14049 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:32.115634 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.116909 29426 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:06:32.119594 14961 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.127854 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.130990 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.131242 14049 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:32.135090 14961 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Bootstrap complete.
I20240502 14:06:32.135327 14961 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.000s
I20240502 14:06:32.135809 14961 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:32.135906 14961 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Initialized, Role: FOLLOWER
I20240502 14:06:32.136024 14961 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:32.136396 14961 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:06:32.139717 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session a7221b90d0aa4770a8ce5522103a96f4-15513e220e144883b216c4f3ac8ade0b received from {username='slave'} at 127.16.217.1:46147
I20240502 14:06:32.139794 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session a7221b90d0aa4770a8ce5522103a96f4-15513e220e144883b216c4f3ac8ade0b on tablet 15513e220e144883b216c4f3ac8ade0b with peer a7221b90d0aa4770a8ce5522103a96f4
I20240502 14:06:32.139855 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.145973 29028 raft_consensus.cc:1203] T f05a1a0bf1314d098f2bf65df58222e2 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:32.170899 15171 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.174332 29425 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.175606 28876 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 8 to 9, VOTER 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7) evicted. New cstate: current_term: 2 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.185178 31178 consensus_queue.cc:237] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.188170 29425 raft_consensus.cc:1261] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:06:32.190161 15171 consensus_queue.cc:1035] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:06:32.192212 14322 raft_consensus.cc:998] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.246s)
W20240502 14:06:32.192349 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cfd44eeeec934413a5001673430e6500. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.192664 14322 consensus_queue.cc:237] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:32.193881 31178 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:32.194335 29159 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:32.194520 30916 raft_consensus.cc:1261] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:32.194988 15171 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.195230 14322 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:32.195525 14322 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:06:32.195751 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:32.196765 29425 raft_consensus.cc:2932] T cfd44eeeec934413a5001673430e6500 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.197328 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet cfd44eeeec934413a5001673430e6500 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:32.197613 14322 consensus_queue.cc:1035] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:32.199304 28876 catalog_manager.cc:5484] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 2 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.206967 15211 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.207849 29159 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.210037 30916 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.210672 28883 catalog_manager.cc:5484] T 5e1b37b67e104154b6755c1762f8f6ff P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.211403 31178 raft_consensus.cc:2932] T 5e1b37b67e104154b6755c1762f8f6ff P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.237644 14482 raft_consensus.cc:998] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.205s)
I20240502 14:06:32.238018 14482 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.241288 30916 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:32.242996 14482 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:32.245661 14482 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.247467 30916 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.249974 28876 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.253757 29159 consensus_queue.cc:237] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:32.254346 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.254988 30916 raft_consensus.cc:1261] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 14:06:32.255780 29095 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573): Couldn't send request to peer c59dc7abff9749abab221ce409d49f59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa944cd0b82429da02bd3dd35e0cce7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.255877 14482 consensus_queue.cc:1035] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:32.258127 14482 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.259399 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4aa944cd0b82429da02bd3dd35e0cce7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:32.259836 30916 raft_consensus.cc:2932] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.261040 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.264132 14482 raft_consensus.cc:998] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.416s)
I20240502 14:06:32.264511 14482 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:32.265352 31178 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:06:32.265667 14482 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:32.267343 28883 catalog_manager.cc:5484] T 4aa944cd0b82429da02bd3dd35e0cce7 P 12348ca540074b32912fdd6333484e0b reported cstate change: config changed from index 8 to 9, VOTER c59dc7abff9749abab221ce409d49f59 (127.16.217.8) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.275665 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.282418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.297433 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.298257 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.300792 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.319134 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.324136 15291 ts_tablet_manager.cc:902] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:32.324611 15291 tablet_copy_client.cc:308] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:06:32.327685 15256 raft_consensus.cc:998] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.465s)
I20240502 14:06:32.328019 15256 consensus_queue.cc:237] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:32.329242 29028 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:06:32.329555 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.329774 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet c88f85f143d845a1b7066244c606f8ba from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:56987)
I20240502 14:06:32.329866 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet c88f85f143d845a1b7066244c606f8ba from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:56987: session id = edce308627b14fa6968d90ec77d51aec-c88f85f143d845a1b7066244c606f8ba
I20240502 14:06:32.330951 29048 tablet_copy_source_session.cc:205] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.331565 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.332139 15291 ts_tablet_manager.cc:1875] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:32.334349 15291 ts_tablet_manager.cc:1888] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 14:06:32.334553 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c88f85f143d845a1b7066244c606f8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.335536 15291 tablet_copy_client.cc:783] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.336395 15256 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:06:32.338665 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.345326 29076 maintenance_manager.cc:419] P a7221b90d0aa4770a8ce5522103a96f4: Scheduling UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b): 4586 bytes on disk
I20240502 14:06:32.348656 28964 maintenance_manager.cc:643] P a7221b90d0aa4770a8ce5522103a96f4: UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 14:06:32.349138 15273 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.351060 15112 raft_consensus.cc:998] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.371s)
I20240502 14:06:32.351362 15112 consensus_queue.cc:237] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.352361 30916 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:32.352778 14288 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:06:32.352883 28883 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 2 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.355019 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 14:06:32.355226 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:32.355686 31178 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.358503 14482 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:32.360486 29027 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:32.361634 29160 consensus_queue.cc:237] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
W20240502 14:06:32.364123 29093 consensus_peers.cc:485] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed713f89a4fd47d79f03711dd14c1334. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.364391 31178 raft_consensus.cc:1261] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:06:32.364763 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.364805 15273 consensus_queue.cc:1035] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:06:32.364815 28876 catalog_manager.cc:5484] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 5 to 6, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:32.367018 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.367139 29159 raft_consensus.cc:1261] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:32.367523 14288 consensus_queue.cc:1035] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:06:32.370682 29160 consensus_queue.cc:237] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:32.376821 15273 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.378248 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ed713f89a4fd47d79f03711dd14c1334 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:32.378952 29027 raft_consensus.cc:1261] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Refusing update from remote peer bd40abfe1fbd48529f9ebbcc4d3abf86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:06:32.379403 15256 consensus_queue.cc:1035] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 14:06:32.379666 29093 consensus_peers.cc:485] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6787832a36648aca61dd26a363089d5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.380347 31178 raft_consensus.cc:2932] T ed713f89a4fd47d79f03711dd14c1334 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.380127 28876 catalog_manager.cc:5484] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 9 to 10, VOTER 12348ca540074b32912fdd6333484e0b (127.16.217.6) added. New cstate: current_term: 2 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.392117 15256 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:32.392776 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.394285 28869 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet e6787832a36648aca61dd26a363089d5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:32.394991 28883 catalog_manager.cc:5484] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86 reported cstate change: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 1 leader_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:32.395397 29027 raft_consensus.cc:2932] T e6787832a36648aca61dd26a363089d5 P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:32.403471 14676 raft_consensus.cc:998] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.297s)
I20240502 14:06:32.403740 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.403748 14676 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
W20240502 14:06:32.404032 29358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 14:06:32.404850 31178 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:06:32.405269 14676 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:32.406646 15291 tablet_copy_client.cc:647] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.410006 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.420361 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.429284 15291 tablet_copy_client.cc:523] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.430835 15291 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:06:32.432618 14288 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.435225 29159 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.435945 30916 raft_consensus.cc:2932] T 785828dee4594ca8813b9418cc5dba2d P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.436064 28876 catalog_manager.cc:5484] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 6 to 7, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:32.448909 15324 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.451020 31178 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.451766 28876 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 7 to 8, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } }
I20240502 14:06:32.461158 15291 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.461236 29425 consensus_queue.cc:237] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:32.461433 15291 tablet_bootstrap.cc:492] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:32.461585 15291 ts_tablet_manager.cc:1367] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.004s
I20240502 14:06:32.461921 15291 raft_consensus.cc:348] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:32.462002 15291 raft_consensus.cc:729] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:32.462107 15291 consensus_queue.cc:260] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:32.462342 15291 ts_tablet_manager.cc:1398] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:06:32.462754 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-c88f85f143d845a1b7066244c606f8ba received from {username='slave'} at 127.16.217.4:56987
I20240502 14:06:32.462831 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session edce308627b14fa6968d90ec77d51aec-c88f85f143d845a1b7066244c606f8ba on tablet c88f85f143d845a1b7066244c606f8ba with peer edce308627b14fa6968d90ec77d51aec
W20240502 14:06:32.471669 29358 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:32.472390 31178 raft_consensus.cc:1261] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:06:32.472995 14676 consensus_queue.cc:1035] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:06:32.476797 29359 consensus_peers.cc:485] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec -> Peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345): Couldn't send request to peer cd692a8958c241fca3aec5cca3a0ded5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:32.476859 14676 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:32.478713 28868 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4b1230a7230f48558da1069a5988b981 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:32.478768 31178 raft_consensus.cc:2932] T 4b1230a7230f48558da1069a5988b981 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:32.480511 28883 catalog_manager.cc:5484] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 8 to 9, VOTER cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) added. New cstate: current_term: 1 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:32.505462 15291 ts_tablet_manager.cc:902] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:32.505949 15291 tablet_copy_client.cc:308] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:06:32.506400 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 81b2d5d0f4044106bed079c3faa72447 from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:56987)
I20240502 14:06:32.506527 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 81b2d5d0f4044106bed079c3faa72447 from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:56987: session id = edce308627b14fa6968d90ec77d51aec-81b2d5d0f4044106bed079c3faa72447
I20240502 14:06:32.507560 29048 tablet_copy_source_session.cc:205] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.508292 15291 ts_tablet_manager.cc:1875] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 14:06:32.508950 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e6787832a36648aca61dd26a363089d5: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:32.510838 15291 ts_tablet_manager.cc:1888] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:06:32.511006 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b2d5d0f4044106bed079c3faa72447. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.511977 15291 tablet_copy_client.cc:783] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.537582 15291 tablet_copy_client.cc:647] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.539252 15291 tablet_copy_client.cc:523] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.540920 15291 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:06:32.553267 15291 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.554375 15291 tablet_bootstrap.cc:492] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:32.554569 15291 ts_tablet_manager.cc:1367] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:06:32.555179 15291 raft_consensus.cc:348] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:32.555286 15291 raft_consensus.cc:729] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:32.556123 15291 consensus_queue.cc:260] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:32.557050 15291 ts_tablet_manager.cc:1398] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:06:32.557515 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-81b2d5d0f4044106bed079c3faa72447 received from {username='slave'} at 127.16.217.4:56987
I20240502 14:06:32.557603 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session edce308627b14fa6968d90ec77d51aec-81b2d5d0f4044106bed079c3faa72447 on tablet 81b2d5d0f4044106bed079c3faa72447 with peer edce308627b14fa6968d90ec77d51aec
I20240502 14:06:32.575027 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.576227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.577586 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.586747 29027 raft_consensus.cc:1203] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 14:06:32.589270 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.590036 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:32.601722 30849 consensus_peers.cc:462] T 400da91cbe234c79859b23ebe1643eb0 P 12348ca540074b32912fdd6333484e0b -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:32.605815 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.620546 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.621449 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.626740 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.636538 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.645424 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.645634 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.659693 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.667984 15488 ts_tablet_manager.cc:902] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:06:32.668535 15488 tablet_copy_client.cc:308] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.8:38573
I20240502 14:06:32.669144 31203 tablet_copy_service.cc:140] P c59dc7abff9749abab221ce409d49f59: Received BeginTabletCopySession request for tablet cfd44eeeec934413a5001673430e6500 from peer 12348ca540074b32912fdd6333484e0b ({username='slave'} at 127.16.217.6:39829)
I20240502 14:06:32.669274 31203 tablet_copy_service.cc:161] P c59dc7abff9749abab221ce409d49f59: Beginning new tablet copy session on tablet cfd44eeeec934413a5001673430e6500 from peer 12348ca540074b32912fdd6333484e0b at {username='slave'} at 127.16.217.6:39829: session id = 12348ca540074b32912fdd6333484e0b-cfd44eeeec934413a5001673430e6500
I20240502 14:06:32.670958 31203 tablet_copy_source_session.cc:205] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.673571 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.676362 15488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfd44eeeec934413a5001673430e6500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.678915 15488 tablet_copy_client.cc:783] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.683794 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447): 4925 bytes on disk
I20240502 14:06:32.684317 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(81b2d5d0f4044106bed079c3faa72447) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 14:06:32.685371 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba): 4840 bytes on disk
I20240502 14:06:32.686085 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(c88f85f143d845a1b7066244c606f8ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20240502 14:06:32.687104 30895 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:32.702160 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.711270 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.714083 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.714888 15488 tablet_copy_client.cc:647] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.716120 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.717783 15488 tablet_copy_client.cc:523] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.719412 15488 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Bootstrap starting.
I20240502 14:06:32.719466 29425 raft_consensus.cc:1203] T c88f85f143d845a1b7066244c606f8ba P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:32.725211 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.726295 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.727553 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:32.734809 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.739229 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.753034 29425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b"
dest_uuid: "edce308627b14fa6968d90ec77d51aec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45414
I20240502 14:06:32.753165 29425 raft_consensus.cc:595] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Received request to transfer leadership
I20240502 14:06:32.785490 15488 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.785810 15488 tablet_bootstrap.cc:492] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Bootstrap complete.
I20240502 14:06:32.786007 15488 ts_tablet_manager.cc:1367] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.004s
I20240502 14:06:32.786444 15488 raft_consensus.cc:348] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.786559 15488 raft_consensus.cc:729] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Initialized, Role: FOLLOWER
I20240502 14:06:32.786705 15488 consensus_queue.cc:260] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.786950 15488 ts_tablet_manager.cc:1398] T cfd44eeeec934413a5001673430e6500 P 12348ca540074b32912fdd6333484e0b: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 14:06:32.788825 31203 tablet_copy_service.cc:342] P c59dc7abff9749abab221ce409d49f59: Request end of tablet copy session 12348ca540074b32912fdd6333484e0b-cfd44eeeec934413a5001673430e6500 received from {username='slave'} at 127.16.217.6:39829
I20240502 14:06:32.788914 31203 tablet_copy_service.cc:433] P c59dc7abff9749abab221ce409d49f59: ending tablet copy session 12348ca540074b32912fdd6333484e0b-cfd44eeeec934413a5001673430e6500 on tablet cfd44eeeec934413a5001673430e6500 with peer 12348ca540074b32912fdd6333484e0b
I20240502 14:06:32.805882 30964 maintenance_manager.cc:419] P 12348ca540074b32912fdd6333484e0b: Scheduling UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500): 4924 bytes on disk
I20240502 14:06:32.806354 30853 maintenance_manager.cc:643] P 12348ca540074b32912fdd6333484e0b: UndoDeltaBlockGCOp(cfd44eeeec934413a5001673430e6500) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 14:06:32.807952 29159 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a"
dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45918
I20240502 14:06:32.808094 29159 raft_consensus.cc:595] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:06:32.819861 29292 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91"
dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55712
I20240502 14:06:32.820011 29292 raft_consensus.cc:595] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:06:32.834345 15580 ts_tablet_manager.cc:902] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:06:32.834944 15580 tablet_copy_client.cc:308] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:06:32.842537 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer c59dc7abff9749abab221ce409d49f59 ({username='slave'} at 127.16.217.8:33673)
I20240502 14:06:32.842650 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet 4aa944cd0b82429da02bd3dd35e0cce7 from peer c59dc7abff9749abab221ce409d49f59 at {username='slave'} at 127.16.217.8:33673: session id = c59dc7abff9749abab221ce409d49f59-4aa944cd0b82429da02bd3dd35e0cce7
I20240502 14:06:32.843703 29181 tablet_copy_source_session.cc:205] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.851250 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa944cd0b82429da02bd3dd35e0cce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.855229 15580 tablet_copy_client.cc:783] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.865036 15601 ts_tablet_manager.cc:902] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Initiating tablet copy from peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347)
I20240502 14:06:32.865893 15601 tablet_copy_client.cc:308] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.4:45347
I20240502 14:06:32.866776 29448 tablet_copy_service.cc:140] P edce308627b14fa6968d90ec77d51aec: Received BeginTabletCopySession request for tablet 4b1230a7230f48558da1069a5988b981 from peer cd692a8958c241fca3aec5cca3a0ded5 ({username='slave'} at 127.16.217.3:45927)
I20240502 14:06:32.866880 29448 tablet_copy_service.cc:161] P edce308627b14fa6968d90ec77d51aec: Beginning new tablet copy session on tablet 4b1230a7230f48558da1069a5988b981 from peer cd692a8958c241fca3aec5cca3a0ded5 at {username='slave'} at 127.16.217.3:45927: session id = cd692a8958c241fca3aec5cca3a0ded5-4b1230a7230f48558da1069a5988b981
I20240502 14:06:32.868116 29448 tablet_copy_source_session.cc:205] T 4b1230a7230f48558da1069a5988b981 P edce308627b14fa6968d90ec77d51aec: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.868921 15601 ts_tablet_manager.cc:1875] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:32.870676 15488 ts_tablet_manager.cc:902] T ed713f89a4fd47d79f03711dd14c1334 P 12348ca540074b32912fdd6333484e0b: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
F20240502 14:06:32.870857 15488 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.queue_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.queue_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_cpu_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_cpu_time_us) (tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_wall_time_us, tablet-download-pool-015b587864d54da8aee36c3afd968e86.run_wall_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.queue_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.queue_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_cpu_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_cpu_time_us) (tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_wall_time_us, tablet-download-pool-0176fb1a25a94c1c9d8cf0dc63577e91.run_wall_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.queue_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_cpu_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_cpu_time_us) (tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_wall_time_us, tablet-download-pool-400da91cbe234c79859b23ebe1643eb0.run_wall_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.queue_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.queue_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_cpu_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_cpu_time_us) (tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_wall_time_us, tablet-download-pool-4aa944cd0b82429da02bd3dd35e0cce7.run_wall_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.queue_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.queue_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_cpu_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_cpu_time_us) (tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_wall_time_us, tablet-download-pool-5a91fd318f0844c1961f16823a7ccc88.run_wall_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.queue_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.queue_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_cpu_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_cpu_time_us) (tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_wall_time_us, tablet-download-pool-5e1b37b67e104154b6755c1762f8f6ff.run_wall_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.queue_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.queue_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_cpu_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_cpu_time_us) (tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_wall_time_us, tablet-download-pool-785828dee4594ca8813b9418cc5dba2d.run_wall_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.queue_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.queue_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_cpu_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_cpu_time_us) (tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_wall_time_us, tablet-download-pool-81b2d5d0f4044106bed079c3faa72447.run_wall_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.queue_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.queue_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_cpu_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_cpu_time_us) (tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_wall_time_us, tablet-download-pool-a476d1664de7456ab21ddbf2e12b4f0d.run_wall_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.queue_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.queue_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_cpu_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_cpu_time_us) (tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_wall_time_us, tablet-download-pool-a75497cf22014c0cb9d079b448284b3d.run_wall_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.queue_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.queue_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_cpu_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_cpu_time_us) (tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_wall_time_us, tablet-download-pool-c88f85f143d845a1b7066244c606f8ba.run_wall_time_us) (tablet-download-pool-cfd44eeeec934413a5001673430e6500.queue_time_us, tablet-download-pool-cfd44eeeec934413a5001673430e6500.queue_time_us) (tablet-download-pool-cfd44eeeec934413a5001673430e6500.run_cpu_time_us, tablet-download-pool-cfd44eeeec934413a5001673430e6500.run_cpu_time_us) (tablet-download-pool-cfd44eeeec934413a5001673430e6500.run_wall_time_us, tablet-download-pool-cfd44eeeec934413a5001673430e6500.run_wall_time_us) (tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.queue_time_us, tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.queue_time_us) (tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.run_cpu_time_us, tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.run_cpu_time_us) (tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.run_wall_time_us, tablet-download-pool-d5ffcf19e66e4adca28d87cbe2245964.run_wall_time_us) (tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.queue_time_us, tablet-download-pool-ed713f89a4fd47d79f03711dd14c1334.queue_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.queue_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.queue_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_cpu_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_cpu_time_us) (tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_wall_time_us, tablet-download-pool-ff65ce0568514e89894ec845fbda31f3.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 14:06:32.872090 15601 ts_tablet_manager.cc:1888] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 14:06:32.872354 15601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1230a7230f48558da1069a5988b981. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f61b0ffedcd google::LogMessage::Fail() at ??:0
I20240502 14:06:32.874267 15601 tablet_copy_client.cc:783] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 5 data blocks...
@ 0x7f61b1002b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f61b0ffe7cc google::LogMessage::Flush() at ??:0
@ 0x7f61b0ffff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:06:32.885713 15601 tablet_copy_client.cc:647] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.888047 15601 tablet_copy_client.cc:523] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f61b2468044 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:06:32.890233 15601 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap starting.
@ 0x7f61b244cf98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:06:32.895181 15580 tablet_copy_client.cc:647] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:32.896618 15580 tablet_copy_client.cc:523] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:32.898111 15580 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Bootstrap starting.
I20240502 14:06:32.898918 15211 raft_consensus.cc:979] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5: : Instructing follower 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:06:32.899012 15211 raft_consensus.cc:1067] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Signalling peer 12348ca540074b32912fdd6333484e0b to start an election
I20240502 14:06:32.899425 30916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91"
dest_uuid: "12348ca540074b32912fdd6333484e0b"
from {username='slave'} at 127.16.217.3:53389
I20240502 14:06:32.899538 30916 raft_consensus.cc:482] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:06:32.899590 30916 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:32.900578 30916 raft_consensus.cc:504] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.901007 30916 leader_election.cc:288] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:06:32.901513 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:06:32.901619 29292 raft_consensus.cc:3032] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:06:32.901660 29292 raft_consensus.cc:729] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:06:32.901782 29292 consensus_queue.cc:260] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.901933 29292 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
@ 0x7f61b244aefc kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 14:06:32.902822 29292 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:06:32.903371 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "12348ca540074b32912fdd6333484e0b" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:06:32.903491 29425 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:32.904631 29425 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12348ca540074b32912fdd6333484e0b in term 2.
I20240502 14:06:32.905068 30850 leader_election.cc:302] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12348ca540074b32912fdd6333484e0b, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:06:32.905241 14498 raft_consensus.cc:2781] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:32.905401 14498 raft_consensus.cc:686] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [term 2 LEADER]: Becoming Leader. State: Replica: 12348ca540074b32912fdd6333484e0b, State: Running, Role: LEADER
@ 0x7f61b4ffd77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:06:32.905547 14498 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:32.909226 28879 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P 12348ca540074b32912fdd6333484e0b reported cstate change: term changed from 1 to 2, leader changed from cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3) to 12348ca540074b32912fdd6333484e0b (127.16.217.6). New cstate: current_term: 2 leader_uuid: "12348ca540074b32912fdd6333484e0b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
@ 0x7f61b5006a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f61b508776f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f61b509b4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:06:32.918934 30895 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
@ 0x7f61b5086226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f61b5095c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ecefd7975c std::function<>::operator()() at ??:0
@ 0x7f61b244fe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61b24506ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f61b245203b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55ecefd7975c std::function<>::operator()() at ??:0
@ 0x7f61b243f807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61b2c7d6db start_thread at ??:0
@ 0x7f61b01b571f clone at ??:0
I20240502 14:06:32.948093 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.951578 15601 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:32.951859 15601 tablet_bootstrap.cc:492] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Bootstrap complete.
I20240502 14:06:32.952021 15601 ts_tablet_manager.cc:1367] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Time spent bootstrapping tablet: real 0.062s user 0.010s sys 0.000s
I20240502 14:06:32.952536 15601 raft_consensus.cc:348] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:32.952656 15601 raft_consensus.cc:729] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Initialized, Role: FOLLOWER
I20240502 14:06:32.952826 15601 consensus_queue.cc:260] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:32.953155 15601 ts_tablet_manager.cc:1398] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:06:32.954121 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.954362 29448 tablet_copy_service.cc:342] P edce308627b14fa6968d90ec77d51aec: Request end of tablet copy session cd692a8958c241fca3aec5cca3a0ded5-4b1230a7230f48558da1069a5988b981 received from {username='slave'} at 127.16.217.3:45927
I20240502 14:06:32.954442 29448 tablet_copy_service.cc:433] P edce308627b14fa6968d90ec77d51aec: ending tablet copy session cd692a8958c241fca3aec5cca3a0ded5-4b1230a7230f48558da1069a5988b981 on tablet 4b1230a7230f48558da1069a5988b981 with peer cd692a8958c241fca3aec5cca3a0ded5
I20240502 14:06:32.957047 15291 ts_tablet_manager.cc:902] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841)
I20240502 14:06:32.957427 15291 tablet_copy_client.cc:308] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.2:35841
I20240502 14:06:32.957791 29181 tablet_copy_service.cc:140] P bd40abfe1fbd48529f9ebbcc4d3abf86: Received BeginTabletCopySession request for tablet e6787832a36648aca61dd26a363089d5 from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:43247)
I20240502 14:06:32.957876 29181 tablet_copy_service.cc:161] P bd40abfe1fbd48529f9ebbcc4d3abf86: Beginning new tablet copy session on tablet e6787832a36648aca61dd26a363089d5 from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:43247: session id = edce308627b14fa6968d90ec77d51aec-e6787832a36648aca61dd26a363089d5
I20240502 14:06:32.958756 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:32.958949 29181 tablet_copy_source_session.cc:205] T e6787832a36648aca61dd26a363089d5 P bd40abfe1fbd48529f9ebbcc4d3abf86: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:32.960223 15291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6787832a36648aca61dd26a363089d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:32.962344 15291 tablet_copy_client.cc:783] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:32.963367 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:32.972913 29006 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:06:32.977144 29273 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:06:32.989729 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:32.992964 29340 maintenance_manager.cc:419] P cd692a8958c241fca3aec5cca3a0ded5: Scheduling UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981): 4586 bytes on disk
I20240502 14:06:32.993503 29229 maintenance_manager.cc:643] P cd692a8958c241fca3aec5cca3a0ded5: UndoDeltaBlockGCOp(4b1230a7230f48558da1069a5988b981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240502 14:06:32.999452 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.007512 15340 raft_consensus.cc:979] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: : Instructing follower a7221b90d0aa4770a8ce5522103a96f4 to start an election
I20240502 14:06:33.007586 15340 raft_consensus.cc:1067] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Signalling peer a7221b90d0aa4770a8ce5522103a96f4 to start an election
I20240502 14:06:33.007977 29027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b"
dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
from {username='slave'} at 127.16.217.4:56987
I20240502 14:06:33.008144 29027 raft_consensus.cc:482] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:06:33.008196 29027 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:33.009207 29027 raft_consensus.cc:504] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:33.009524 29027 leader_election.cc:288] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 election: Requested vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573)
I20240502 14:06:33.010200 31178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "c59dc7abff9749abab221ce409d49f59"
I20240502 14:06:33.010339 31178 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:33.011230 31178 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 3.
I20240502 14:06:33.011509 28962 leader_election.cc:302] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, c59dc7abff9749abab221ce409d49f59; no voters:
I20240502 14:06:33.011660 15112 raft_consensus.cc:2781] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:06:33.011796 15112 raft_consensus.cc:686] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Becoming Leader. State: Replica: a7221b90d0aa4770a8ce5522103a96f4, State: Running, Role: LEADER
I20240502 14:06:33.011907 15112 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:33.012913 29425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15513e220e144883b216c4f3ac8ade0b" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "edce308627b14fa6968d90ec77d51aec"
I20240502 14:06:33.013023 29425 raft_consensus.cc:3032] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:06:33.013065 29425 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:06:33.013218 29425 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:33.013370 29425 raft_consensus.cc:3037] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:33.013664 28876 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: term changed from 2 to 3, leader changed from edce308627b14fa6968d90ec77d51aec (127.16.217.4) to a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1). New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:33.014281 29425 raft_consensus.cc:2454] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 3.
I20240502 14:06:33.017405 15291 tablet_copy_client.cc:647] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:33.021260 15291 tablet_copy_client.cc:523] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:33.023046 15291 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:06:33.029939 15580 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:33.030226 15580 tablet_bootstrap.cc:492] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Bootstrap complete.
I20240502 14:06:33.030424 15580 ts_tablet_manager.cc:1367] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Time spent bootstrapping tablet: real 0.132s user 0.009s sys 0.000s
I20240502 14:06:33.030827 15580 raft_consensus.cc:348] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:33.030922 15580 raft_consensus.cc:729] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Initialized, Role: FOLLOWER
I20240502 14:06:33.036263 15580 consensus_queue.cc:260] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:06:33.036674 15580 ts_tablet_manager.cc:1398] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:06:33.037597 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session c59dc7abff9749abab221ce409d49f59-4aa944cd0b82429da02bd3dd35e0cce7 received from {username='slave'} at 127.16.217.8:33673
I20240502 14:06:33.037676 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session c59dc7abff9749abab221ce409d49f59-4aa944cd0b82429da02bd3dd35e0cce7 on tablet 4aa944cd0b82429da02bd3dd35e0cce7 with peer c59dc7abff9749abab221ce409d49f59
I20240502 14:06:33.043828 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.047837 15291 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:33.048511 15291 tablet_bootstrap.cc:492] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:33.048691 15291 ts_tablet_manager.cc:1367] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.002s
I20240502 14:06:33.049057 15291 raft_consensus.cc:348] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:33.049172 15291 raft_consensus.cc:729] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:33.049309 15291 consensus_queue.cc:260] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:33.049647 15291 ts_tablet_manager.cc:1398] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:06:33.050588 29181 tablet_copy_service.cc:342] P bd40abfe1fbd48529f9ebbcc4d3abf86: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-e6787832a36648aca61dd26a363089d5 received from {username='slave'} at 127.16.217.4:43247
I20240502 14:06:33.050671 29181 tablet_copy_service.cc:433] P bd40abfe1fbd48529f9ebbcc4d3abf86: ending tablet copy session edce308627b14fa6968d90ec77d51aec-e6787832a36648aca61dd26a363089d5 on tablet e6787832a36648aca61dd26a363089d5 with peer edce308627b14fa6968d90ec77d51aec
I20240502 14:06:33.055328 29273 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:06:33.065738 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.065752 29141 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:33.068943 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:33.072849 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.083408 15273 raft_consensus.cc:979] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: : Instructing follower cd692a8958c241fca3aec5cca3a0ded5 to start an election
I20240502 14:06:33.083480 15273 raft_consensus.cc:1067] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Signalling peer cd692a8958c241fca3aec5cca3a0ded5 to start an election
I20240502 14:06:33.083882 29292 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a"
dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
from {username='slave'} at 127.16.217.2:59669
I20240502 14:06:33.084028 29292 raft_consensus.cc:482] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:06:33.084086 29292 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:33.085202 29292 raft_consensus.cc:504] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:33.085557 29292 leader_election.cc:288] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 2 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:33.086319 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4"
I20240502 14:06:33.086437 29027 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:33.087458 29027 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:06:33.087720 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dda4193aaaf459fb5049ff49541229a" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:06:33.087829 29159 raft_consensus.cc:3032] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:06:33.087867 29159 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Running, Role: LEADER
I20240502 14:06:33.088016 29159 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:33.088181 29159 raft_consensus.cc:3037] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:06:33.088505 29227 leader_election.cc:302] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:06:33.088876 15642 raft_consensus.cc:2781] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:06:33.089032 15642 raft_consensus.cc:686] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 2 LEADER]: Becoming Leader. State: Replica: cd692a8958c241fca3aec5cca3a0ded5, State: Running, Role: LEADER
I20240502 14:06:33.089169 15642 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:33.089551 29159 raft_consensus.cc:2454] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
I20240502 14:06:33.091123 28879 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 reported cstate change: term changed from 1 to 2, leader changed from bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) to cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3). New cstate: current_term: 2 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:33.095029 29425 raft_consensus.cc:1203] T 81b2d5d0f4044106bed079c3faa72447 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:33.096458 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.128368 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.135380 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.157485 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.162276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.181366 31230 maintenance_manager.cc:419] P c59dc7abff9749abab221ce409d49f59: Scheduling UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7): 4925 bytes on disk
I20240502 14:06:33.181879 31119 maintenance_manager.cc:643] P c59dc7abff9749abab221ce409d49f59: UndoDeltaBlockGCOp(4aa944cd0b82429da02bd3dd35e0cce7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 14:06:33.186197 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.188637 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5): 4840 bytes on disk
I20240502 14:06:33.188674 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.188977 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(e6787832a36648aca61dd26a363089d5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 14:06:33.233966 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.242019 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.243659 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.272552 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.273938 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:33.294786 29225 connection.cc:535] client connection to 127.16.217.6:38737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:06:33.294891 29093 connection.cc:535] client connection to 127.16.217.6:38737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:06:33.294960 31115 connection.cc:535] client connection to 127.16.217.6:38737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:06:33.298444 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.298534 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.298583 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.299787 28960 connection.cc:535] client connection to 127.16.217.6:38737 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:06:33.303603 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.303673 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.303717 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.303761 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:33.316092 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:33.316174 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.325939 31233 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhNHehp/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:01:19.820607 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.820905 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.826086 31240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:01:20.787317 31240 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:01:21.235749 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.235853 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.235888 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.235944 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.235980 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.236052 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.236128 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.236161 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.236220 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.236250 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.236294 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.236347 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.236394 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.238000 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
I20240502 14:01:21.331949 31240 rebalancer_tool.cc:1074] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:21.378389 31240 rebalancer_tool.cc:1074] tablet f05a1a0bf1314d098f2bf65df58222e2: 'a7221b90d0aa4770a8ce5522103a96f4' -> '33ebf308f6f24e6e8768ef5f14ddc785' move scheduled
I20240502 14:01:21.429236 31240 rebalancer_tool.cc:1074] tablet 2b1adebdbd104796b927963e86a15295: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:21.556486 31240 rebalancer_tool.cc:1074] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:21.629057 31240 rebalancer_tool.cc:1074] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:21.682508 31240 rebalancer_tool.cc:1074] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'a7221b90d0aa4770a8ce5522103a96f4' -> '33ebf308f6f24e6e8768ef5f14ddc785' move scheduled
I20240502 14:01:22.625021 31240 rebalancer_tool.cc:1074] tablet ff65ce0568514e89894ec845fbda31f3: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:22.836864 31240 rebalancer_tool.cc:1074] tablet db68071f838a4e209f3db69bea6e54f3: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:22.881673 31240 rebalancer_tool.cc:1074] tablet a476d1664de7456ab21ddbf2e12b4f0d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:22.913273 31240 rebalancer_tool.cc:1084] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'a7221b90d0aa4770a8ce5522103a96f4' -> '33ebf308f6f24e6e8768ef5f14ddc785' move ignored: Illegal state: tablet 86cfc7d5709f414aac098ae3c05a48a3: replica 33ebf308f6f24e6e8768ef5f14ddc785 is already present
I20240502 14:01:22.913352 31240 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:23.402854 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.402927 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:23.402957 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.403046 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.403082 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:23.403115 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:23.403147 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:23.403180 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:23.403244 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:23.403281 31240 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:23.425488 31240 rebalancer_tool.cc:1074] tablet ed713f89a4fd47d79f03711dd14c1334: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:23.433312 31240 rebalancer_tool.cc:1074] tablet a75497cf22014c0cb9d079b448284b3d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:23.459075 31240 rebalancer_tool.cc:1074] tablet c88f85f143d845a1b7066244c606f8ba: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:23.505754 31240 rebalancer_tool.cc:1074] tablet cfd44eeeec934413a5001673430e6500: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:23.548485 31240 rebalancer_tool.cc:1074] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:25.557703 31240 rebalancer_tool.cc:1123] tablet ed713f89a4fd47d79f03711dd14c1334: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move is abandoned: Remote error: Service unavailable: leader transfer in progress
I20240502 14:01:26.019327 31240 rebalancer_tool.cc:1136] tablet ff65ce0568514e89894ec845fbda31f3: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:26.596035 31240 rebalancer_tool.cc:1136] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:26.626323 31240 rebalancer_tool.cc:1136] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'a7221b90d0aa4770a8ce5522103a96f4' -> '33ebf308f6f24e6e8768ef5f14ddc785' move completed: OK
I20240502 14:01:27.748972 31240 rebalancer_tool.cc:1136] tablet 2b1adebdbd104796b927963e86a15295: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: OK
I20240502 14:01:27.759330 31240 rebalancer_tool.cc:1136] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:27.766765 31240 rebalancer_tool.cc:1136] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:27.775163 31240 rebalancer_tool.cc:1136] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move completed: OK
I20240502 14:01:27.784966 31240 rebalancer_tool.cc:1136] tablet cfd44eeeec934413a5001673430e6500: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:27.816989 31240 rebalancer_tool.cc:1136] tablet a75497cf22014c0cb9d079b448284b3d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:27.965194 31240 rebalancer_tool.cc:1136] tablet db68071f838a4e209f3db69bea6e54f3: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: OK
I20240502 14:01:27.996445 31240 rebalancer_tool.cc:1136] tablet f05a1a0bf1314d098f2bf65df58222e2: 'a7221b90d0aa4770a8ce5522103a96f4' -> '33ebf308f6f24e6e8768ef5f14ddc785' move completed: OK
I20240502 14:01:28.024358 31240 rebalancer_tool.cc:1136] tablet c88f85f143d845a1b7066244c606f8ba: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:28.560458 31240 rebalancer_tool.cc:1074] tablet 1dda4193aaaf459fb5049ff49541229a: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:28.573002 31240 rebalancer_tool.cc:1074] tablet 400da91cbe234c79859b23ebe1643eb0: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:28.593502 31240 rebalancer_tool.cc:1136] tablet 400da91cbe234c79859b23ebe1643eb0: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:28.625391 31240 rebalancer_tool.cc:1136] tablet 1dda4193aaaf459fb5049ff49541229a: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:28.753146 31240 rebalancer_tool.cc:1136] tablet a476d1664de7456ab21ddbf2e12b4f0d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:29.357960 31240 rebalancer_tool.cc:1074] tablet 785828dee4594ca8813b9418cc5dba2d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:29.423789 31240 rebalancer_tool.cc:1074] tablet d5ffcf19e66e4adca28d87cbe2245964: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:29.514609 31240 rebalancer_tool.cc:1074] tablet 4b1230a7230f48558da1069a5988b981: 'cd692a8958c241fca3aec5cca3a0ded5' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:29.635205 31240 rebalancer_tool.cc:1074] tablet 81b2d5d0f4044106bed079c3faa72447: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:29.649006 31240 rebalancer_tool.cc:1074] tablet db68071f838a4e209f3db69bea6e54f3: '33ebf308f6f24e6e8768ef5f14ddc785' -> '' move scheduled
I20240502 14:01:29.657064 31240 rebalancer_tool.cc:1074] tablet 15513e220e144883b216c4f3ac8ade0b: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:29.663762 31240 rebalancer_tool.cc:1074] tablet f05a1a0bf1314d098f2bf65df58222e2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
W20240502 14:01:30.990581 5560 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:30.941229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.7:40969 (local address 127.0.0.1:59098)
0502 14:01:30.948817 (+ 7588us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:30.948830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:30.948895 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:30.989967 (+ 41072us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":7531,"thread_start_us":56,"threads_started":1}
W20240502 14:01:32.323086 6396 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:32.252295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.5:45769 (local address 127.0.0.1:46792)
0502 14:01:32.252785 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:32.252800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:32.252885 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:32.322989 (+ 70104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":77,"threads_started":1}
I20240502 14:01:33.042271 31240 rebalancer_tool.cc:1136] tablet f05a1a0bf1314d098f2bf65df58222e2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:33.052033 31240 rebalancer_tool.cc:1136] tablet 15513e220e144883b216c4f3ac8ade0b: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:33.386699 31240 rebalancer_tool.cc:1136] tablet 81b2d5d0f4044106bed079c3faa72447: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:33.396061 31240 rebalancer_tool.cc:1136] tablet 4b1230a7230f48558da1069a5988b981: 'cd692a8958c241fca3aec5cca3a0ded5' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:33.437366 31240 rebalancer_tool.cc:1136] tablet d5ffcf19e66e4adca28d87cbe2245964: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
W20240502 14:03:36.670512 12483 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ListTables -> {remote=127.16.217.62:41117, user_credentials={real_user=slave}} blocked reactor thread for 45970.4us
W20240502 14:05:53.201314 25100 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.000s sys 0.000s
I20240502 14:06:31.321648 31240 rebalancer_tool.cc:1136] tablet db68071f838a4e209f3db69bea6e54f3: '33ebf308f6f24e6e8768ef5f14ddc785' -> '' move completed: OK
I20240502 14:06:32.666191 31240 rebalancer_tool.cc:1136] tablet 785828dee4594ca8813b9418cc5dba2d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: Incomplete: tablet 785828dee4594ca8813b9418cc5dba2d, TS bd40abfe1fbd48529f9ebbcc4d3abf86 -> TS 22dbbf207ba0440392e71a10c2ca78ca move failed, target replica disappeared
Illegal state: tablet server 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): unacceptable health status UNAVAILABLE
I20240502 14:06:33.353435 29292 raft_consensus.cc:1203] T 4b1230a7230f48558da1069a5988b981 P cd692a8958c241fca3aec5cca3a0ded5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 14:06:33.376479 31178 raft_consensus.cc:1203] T 4aa944cd0b82429da02bd3dd35e0cce7 P c59dc7abff9749abab221ce409d49f59 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240502 14:06:33.378012 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.383098 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.384945 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.387367 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.388396 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.394862 28873 maintenance_manager.cc:419] P db037a839b724e0babeaf39df2515fc8: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7001 bytes on disk
I20240502 14:06:33.397586 28872 maintenance_manager.cc:643] P db037a839b724e0babeaf39df2515fc8: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"thread_start_us":64,"threads_started":1}
I20240502 14:06:33.403120 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.470940 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:06:33.471664 15683 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:06:33.476106 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.479641 29425 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:06:33.480087 15683 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:06:33.484211 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.485630 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.486878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.487188 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.511075 15682 raft_consensus.cc:998] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec: Attempting to remove follower 22dbbf207ba0440392e71a10c2ca78ca from the Raft config. Reason: Leader has been unable to successfully communicate with peer 22dbbf207ba0440392e71a10c2ca78ca for more than 300 seconds (300.271s)
I20240502 14:06:33.511456 15682 consensus_queue.cc:237] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:33.512912 29159 raft_consensus.cc:1261] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
W20240502 14:06:33.513723 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.513890 15682 consensus_queue.cc:1035] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:33.516721 15682 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 6 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:33.519001 29159 raft_consensus.cc:2932] T 015b587864d54da8aee36c3afd968e86 P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 6 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:33.519569 28878 catalog_manager.cc:5484] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec reported cstate change: config changed from index 6 to 9, VOTER 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5) evicted. New cstate: current_term: 3 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
W20240502 14:06:33.524040 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:33.566502 29426 raft_consensus.cc:1203] T e6787832a36648aca61dd26a363089d5 P edce308627b14fa6968d90ec77d51aec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 14:06:33.583248 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:06:33.587440 29027 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:06:33.588119 15642 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:33.597826 29159 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:06:33.598253 15642 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:06:33.621122 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.630784 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.634081 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.637049 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.637303 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.654071 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.654495 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.657771 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.676240 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.682447 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.699680 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.709427 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.709904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.719718 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.723968 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.774463 29292 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } }
I20240502 14:06:33.776063 29027 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:06:33.776525 15986 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:33.778148 15986 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:33.781927 29027 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:33.784895 28881 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 7 to 9, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) evicted. New cstate: current_term: 2 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } }
I20240502 14:06:33.786975 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.787948 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.788616 29141 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dda4193aaaf459fb5049ff49541229a with delete_type TABLET_DATA_TOMBSTONED (TS bd40abfe1fbd48529f9ebbcc4d3abf86 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45670
I20240502 14:06:33.789007 29292 consensus_queue.cc:237] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:33.790040 16088 tablet_replica.cc:330] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: stopping tablet replica
I20240502 14:06:33.790319 16088 raft_consensus.cc:2229] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:33.790467 16088 raft_consensus.cc:2258] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 14:06:33.790882 29225 consensus_peers.cc:485] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:33.791379 29027 raft_consensus.cc:1261] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Refusing update from remote peer cd692a8958c241fca3aec5cca3a0ded5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:06:33.791738 15986 consensus_queue.cc:1035] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:06:33.793352 15986 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.793505 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.795039 29028 raft_consensus.cc:2932] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.796986 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 1dda4193aaaf459fb5049ff49541229a with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:33.800810 28878 catalog_manager.cc:5484] T 1dda4193aaaf459fb5049ff49541229a P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 9 to 10, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 2 leader_uuid: "cd692a8958c241fca3aec5cca3a0ded5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.802863 16088 ts_tablet_manager.cc:1875] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:33.811554 16088 ts_tablet_manager.cc:1888] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 14:06:33.811733 16088 log.cc:1196] T 1dda4193aaaf459fb5049ff49541229a P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-1/wal/wals/1dda4193aaaf459fb5049ff49541229a
I20240502 14:06:33.812338 28869 catalog_manager.cc:4830] TS bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): tablet 1dda4193aaaf459fb5049ff49541229a (table rebalance_test_table_5 [id=6e061e16779c475bbfadb37b5432fc9f]) successfully deleted
I20240502 14:06:33.814267 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.815212 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.824626 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:33.826108 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 14:06:33.826521 15683 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:06:33.829747 15683 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:33.832073 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:33.832194 28881 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 10 to 12, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:33.836153 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:33.837167 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 15513e220e144883b216c4f3ac8ade0b with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:45210
I20240502 14:06:33.837276 29426 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:06:33.837781 15112 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 14:06:33.839048 29426 raft_consensus.cc:858] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
W20240502 14:06:33.839116 29426 raft_consensus.cc:1565] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 NON_PARTICIPANT]: Could not prepare op 'term: 3 index: 13' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:33.839439 29426 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 NON_PARTICIPANT]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:33.842847 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 14:06:33.843202 29426 raft_consensus.cc:1467] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 14:06:33.843216 15683 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 14:06:33.844252 15683 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.846243 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.846455 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 15513e220e144883b216c4f3ac8ade0b with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:33.848896 28881 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 12 to 13, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:33.849587 29426 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
W20240502 14:06:33.853456 28868 catalog_manager.cc:4804] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): delete failed for tablet 15513e220e144883b216c4f3ac8ade0b because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 10 but the committed config has opid_index of 13
I20240502 14:06:33.887655 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.888803 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.899200 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.899398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:33.899577 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 14:06:33.899933 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:33.959267 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:33.961640 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:33.961803 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:33.969240 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:33.983721 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.063158 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.063637 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:34.065729 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:34.065732 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.066228 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:34.112433 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:06:34.126622 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.128520 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:34.134125 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:34.134821 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.151392 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.170464 15986 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.170573 15986 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.170987 15986 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c59dc7abff9749abab221ce409d49f59 (127.16.217.8:38573), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.171676 31178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "c59dc7abff9749abab221ce409d49f59" is_pre_election: true
W20240502 14:06:34.172683 29225 leader_election.cc:334] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.172755 29225 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5; no voters: 12348ca540074b32912fdd6333484e0b, c59dc7abff9749abab221ce409d49f59
I20240502 14:06:34.172920 15986 raft_consensus.cc:2726] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:34.202410 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.204401 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.205915 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:34.214017 31234 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhNHehp/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:01:19.753165 31241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.753397 31241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.769112 31241 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:01:20.778750 31241 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:01:21.277755 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.277879 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.277933 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.277992 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.278110 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.278203 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.278311 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.278357 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.278395 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.278434 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.278471 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.278508 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.278545 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.278584 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
I20240502 14:01:21.361950 31241 rebalancer_tool.cc:1074] tablet 5e1b37b67e104154b6755c1762f8f6ff: 'a7221b90d0aa4770a8ce5522103a96f4' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:21.444679 31241 rebalancer_tool.cc:1074] tablet 81b2d5d0f4044106bed079c3faa72447: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:21.517006 31241 rebalancer_tool.cc:1084] tablet dcc536794ffb453495d025f29ea152bc: 'cd692a8958c241fca3aec5cca3a0ded5' -> '33ebf308f6f24e6e8768ef5f14ddc785' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } }
I20240502 14:01:21.517107 31241 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:21.893041 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893149 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893215 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.893256 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.893296 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.893337 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.893378 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893433 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.893491 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.893535 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893579 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.893620 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.893656 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.893694 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893733 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.893772 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:21.893808 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.893850 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.893894 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:01:21.972369 31241 rebalancer_tool.cc:1074] tablet 15513e220e144883b216c4f3ac8ade0b: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:22.168373 31241 rebalancer_tool.cc:1074] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'edce308627b14fa6968d90ec77d51aec' -> '33ebf308f6f24e6e8768ef5f14ddc785' move scheduled
I20240502 14:01:22.297613 31241 rebalancer_tool.cc:1074] tablet e6787832a36648aca61dd26a363089d5: 'cd692a8958c241fca3aec5cca3a0ded5' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:22.381544 31241 rebalancer_tool.cc:1074] tablet d5ffcf19e66e4adca28d87cbe2245964: 'a7221b90d0aa4770a8ce5522103a96f4' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:22.390882 31241 rebalancer_tool.cc:1084] tablet a476d1664de7456ab21ddbf2e12b4f0d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '33ebf308f6f24e6e8768ef5f14ddc785' move ignored: Illegal state: tablet a476d1664de7456ab21ddbf2e12b4f0d: replica 33ebf308f6f24e6e8768ef5f14ddc785 is already present
I20240502 14:01:22.390954 31241 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:22.811554 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:22.811645 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:22.811691 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.811756 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.811800 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:22.811841 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.811878 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.811992 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.812036 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:22.812083 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.812124 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.844846 31241 rebalancer_tool.cc:1074] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:22.908878 31241 rebalancer_tool.cc:1074] tablet 20e960d4e55047918a1dad82fae21ecd: 'a7221b90d0aa4770a8ce5522103a96f4' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:22.980800 31241 rebalancer_tool.cc:1074] tablet d9e0261898a445949f3cf21d2efb599f: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '33ebf308f6f24e6e8768ef5f14ddc785' move scheduled
I20240502 14:01:22.993999 31241 rebalancer_tool.cc:1074] tablet ed713f89a4fd47d79f03711dd14c1334: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:23.096097 31241 rebalancer_tool.cc:1084] tablet 4aa944cd0b82429da02bd3dd35e0cce7: 'a7221b90d0aa4770a8ce5522103a96f4' -> '22dbbf207ba0440392e71a10c2ca78ca' move ignored: Illegal state: tablet 4aa944cd0b82429da02bd3dd35e0cce7: replica 22dbbf207ba0440392e71a10c2ca78ca is already present
I20240502 14:01:23.096199 31241 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:23.488400 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.488504 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:23.488615 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:23.488695 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:23.488737 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.488775 31241 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
I20240502 14:01:23.540087 31241 rebalancer_tool.cc:1074] tablet a75497cf22014c0cb9d079b448284b3d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:23.565976 31241 rebalancer_tool.cc:1074] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:23.589231 31241 rebalancer_tool.cc:1074] tablet 400da91cbe234c79859b23ebe1643eb0: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:27.717381 31241 rebalancer_tool.cc:1136] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:27.724835 31241 rebalancer_tool.cc:1136] tablet 81b2d5d0f4044106bed079c3faa72447: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: OK
I20240502 14:01:28.982342 31241 rebalancer_tool.cc:1136] tablet 5e1b37b67e104154b6755c1762f8f6ff: 'a7221b90d0aa4770a8ce5522103a96f4' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:29.971124 31241 rebalancer_tool.cc:1074] tablet 19784717201a464c84607d2a43e651c2: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:30.149109 31241 rebalancer_tool.cc:1136] tablet 19784717201a464c84607d2a43e651c2: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:30.169893 31241 rebalancer_tool.cc:1136] tablet 400da91cbe234c79859b23ebe1643eb0: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move completed: OK
I20240502 14:01:30.198688 31241 rebalancer_tool.cc:1136] tablet a75497cf22014c0cb9d079b448284b3d: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:30.875654 31241 rebalancer_tool.cc:1136] tablet d9e0261898a445949f3cf21d2efb599f: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '33ebf308f6f24e6e8768ef5f14ddc785' move completed: OK
I20240502 14:01:30.884440 31241 rebalancer_tool.cc:1136] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:30.915452 31241 rebalancer_tool.cc:1136] tablet 20e960d4e55047918a1dad82fae21ecd: 'a7221b90d0aa4770a8ce5522103a96f4' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:31.532845 31241 rebalancer_tool.cc:1136] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'edce308627b14fa6968d90ec77d51aec' -> '33ebf308f6f24e6e8768ef5f14ddc785' move completed: OK
I20240502 14:01:31.561276 31241 rebalancer_tool.cc:1136] tablet e6787832a36648aca61dd26a363089d5: 'cd692a8958c241fca3aec5cca3a0ded5' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: OK
I20240502 14:01:31.600226 31241 rebalancer_tool.cc:1136] tablet d5ffcf19e66e4adca28d87cbe2245964: 'a7221b90d0aa4770a8ce5522103a96f4' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:31.607885 31241 rebalancer_tool.cc:1074] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:31.629266 31241 rebalancer_tool.cc:1074] tablet cfd44eeeec934413a5001673430e6500: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:31.639945 31241 rebalancer_tool.cc:1074] tablet c88f85f143d845a1b7066244c606f8ba: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:31.655745 31241 rebalancer_tool.cc:1074] tablet 2b1adebdbd104796b927963e86a15295: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
W20240502 14:01:32.324151 6121 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:01:32.318979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.5:45769 (local address 127.0.0.1:46818)
0502 14:01:32.319153 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0502 14:01:32.319163 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0502 14:01:32.319226 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:01:32.323405 (+ 4179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":84}
I20240502 14:01:32.665437 31241 rebalancer_tool.cc:1136] tablet c88f85f143d845a1b7066244c606f8ba: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:32.680660 31241 rebalancer_tool.cc:1136] tablet cfd44eeeec934413a5001673430e6500: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:33.199775 31241 rebalancer_tool.cc:1136] tablet ed713f89a4fd47d79f03711dd14c1334: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:33.278043 31241 rebalancer_tool.cc:1136] tablet 15513e220e144883b216c4f3ac8ade0b: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:33.338414 31241 rebalancer_tool.cc:1074] tablet 1dda4193aaaf459fb5049ff49541229a: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:33.368799 31241 rebalancer_tool.cc:1074] tablet f05a1a0bf1314d098f2bf65df58222e2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:33.382645 31241 rebalancer_tool.cc:1074] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:33.390900 31241 rebalancer_tool.cc:1136] tablet f05a1a0bf1314d098f2bf65df58222e2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:33.421561 31241 rebalancer_tool.cc:1136] tablet 1dda4193aaaf459fb5049ff49541229a: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:33.428223 31241 rebalancer_tool.cc:1136] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
W20240502 14:01:37.352890 9305 meta_cache.cc:1261] Time spent looking up entry by key: real 0.068s user 0.001s sys 0.000s
W20240502 14:05:46.147832 21195 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.16.217.62:41117, user_credentials={real_user=slave}} blocked reactor thread for 47194.7us
W20240502 14:06:04.936069 32546 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.16.217.62:41117, user_credentials={real_user=slave}} blocked reactor thread for 50517.4us
W20240502 14:06:33.295579 15579 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:33.225403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.6:38737 (local address 127.0.0.1:40824)
0502 14:06:33.225578 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:33.225591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:33.225691 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:33.295466 (+ 69775us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":96}
I20240502 14:06:34.006132 31241 rebalancer_tool.cc:1123] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.006210 31241 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): unacceptable health status UNAVAILABLE
I20240502 14:06:34.217609 15994 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.217725 15994 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.218133 15994 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.218957 29426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "edce308627b14fa6968d90ec77d51aec" is_pre_election: true
W20240502 14:06:34.220036 29093 leader_election.cc:334] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.220124 29093 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86; no voters: 12348ca540074b32912fdd6333484e0b, edce308627b14fa6968d90ec77d51aec
I20240502 14:06:34.220414 15994 raft_consensus.cc:2726] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:34.234584 31161 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:06:34.235319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.251277 15976 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.251392 15976 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.251857 15976 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
W20240502 14:06:34.256429 31115 leader_election.cc:334] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.257483 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "c59dc7abff9749abab221ce409d49f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:34.257622 29292 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59dc7abff9749abab221ce409d49f59 in term 2.
I20240502 14:06:34.257913 31117 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c59dc7abff9749abab221ce409d49f59, cd692a8958c241fca3aec5cca3a0ded5; no voters: 12348ca540074b32912fdd6333484e0b
I20240502 14:06:34.258147 15976 raft_consensus.cc:2781] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:06:34.258200 15976 raft_consensus.cc:482] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.258246 15976 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:34.259387 15976 raft_consensus.cc:504] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.259757 15976 leader_election.cc:288] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.260656 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffcf19e66e4adca28d87cbe2245964" candidate_uuid: "c59dc7abff9749abab221ce409d49f59" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:06:34.260751 29292 raft_consensus.cc:3037] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:34.261646 29292 raft_consensus.cc:2454] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59dc7abff9749abab221ce409d49f59 in term 3.
I20240502 14:06:34.261907 31117 leader_election.cc:302] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c59dc7abff9749abab221ce409d49f59, cd692a8958c241fca3aec5cca3a0ded5; no voters:
I20240502 14:06:34.262264 15976 raft_consensus.cc:2781] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:06:34.262410 15976 raft_consensus.cc:686] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [term 3 LEADER]: Becoming Leader. State: Replica: c59dc7abff9749abab221ce409d49f59, State: Running, Role: LEADER
I20240502 14:06:34.262549 15976 consensus_queue.cc:237] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.264045 28880 catalog_manager.cc:5484] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 reported cstate change: term changed from 2 to 3, leader changed from 12348ca540074b32912fdd6333484e0b (127.16.217.6) to c59dc7abff9749abab221ce409d49f59 (127.16.217.8). New cstate: current_term: 3 leader_uuid: "c59dc7abff9749abab221ce409d49f59" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
W20240502 14:06:34.265005 31115 leader_election.cc:334] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.280779 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.285540 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:34.285662 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.287375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.288446 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.302791 16381 ts_tablet_manager.cc:902] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345)
I20240502 14:06:34.303278 16381 tablet_copy_client.cc:308] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.3:41345
I20240502 14:06:34.303702 29313 tablet_copy_service.cc:140] P cd692a8958c241fca3aec5cca3a0ded5: Received BeginTabletCopySession request for tablet 1dda4193aaaf459fb5049ff49541229a from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:37763)
I20240502 14:06:34.303799 29313 tablet_copy_service.cc:161] P cd692a8958c241fca3aec5cca3a0ded5: Beginning new tablet copy session on tablet 1dda4193aaaf459fb5049ff49541229a from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:37763: session id = edce308627b14fa6968d90ec77d51aec-1dda4193aaaf459fb5049ff49541229a
I20240502 14:06:34.307147 29313 tablet_copy_source_session.cc:205] T 1dda4193aaaf459fb5049ff49541229a P cd692a8958c241fca3aec5cca3a0ded5: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:34.307994 16381 ts_tablet_manager.cc:1875] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:34.312225 16381 ts_tablet_manager.cc:1888] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 14:06:34.312404 16381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dda4193aaaf459fb5049ff49541229a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:34.313338 16381 tablet_copy_client.cc:783] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:34.316377 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:34.317746 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:34.319000 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 14:06:34.319600 16110 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 14:06:34.321367 16110 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:34.324035 28884 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 13 to 14, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:34.324242 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:34.328152 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 15513e220e144883b216c4f3ac8ade0b with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:45210
I20240502 14:06:34.328408 16109 tablet_replica.cc:330] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:06:34.328531 16109 raft_consensus.cc:2229] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:34.328648 16109 raft_consensus.cc:2258] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:34.328668 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:34.329543 16109 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:34.330320 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
W20240502 14:06:34.330379 28960 consensus_peers.cc:485] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 -> Peer edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): Couldn't send request to peer edce308627b14fa6968d90ec77d51aec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:34.330797 15683 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240502 14:06:34.331537 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 14:06:34.332700 15112 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:34.332839 16109 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 14:06:34.332942 16109 log.cc:1196] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/15513e220e144883b216c4f3ac8ade0b
I20240502 14:06:34.333108 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:34.335932 16381 tablet_copy_client.cc:647] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:34.336686 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 15513e220e144883b216c4f3ac8ade0b (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:06:34.337535 28880 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 reported cstate change: config changed from index 14 to 15, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) added. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } }
I20240502 14:06:34.337982 16381 tablet_copy_client.cc:523] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:34.338321 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 15513e220e144883b216c4f3ac8ade0b with cas_config_opid_index 14: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:34.338441 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.341892 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.341928 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.349017 16381 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
I20240502 14:06:34.350900 29403 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:06:34.363154 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:34.372367 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:06:34.373248 16381 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:34.373526 16381 tablet_bootstrap.cc:492] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:34.373694 16381 ts_tablet_manager.cc:1367] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240502 14:06:34.374183 16381 raft_consensus.cc:348] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:34.374294 16381 raft_consensus.cc:729] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:34.374425 16381 consensus_queue.cc:260] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:34.374764 16381 ts_tablet_manager.cc:1398] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:06:34.375232 29313 tablet_copy_service.cc:342] P cd692a8958c241fca3aec5cca3a0ded5: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-1dda4193aaaf459fb5049ff49541229a received from {username='slave'} at 127.16.217.4:37763
I20240502 14:06:34.375306 29313 tablet_copy_service.cc:433] P cd692a8958c241fca3aec5cca3a0ded5: ending tablet copy session edce308627b14fa6968d90ec77d51aec-1dda4193aaaf459fb5049ff49541229a on tablet 1dda4193aaaf459fb5049ff49541229a with peer edce308627b14fa6968d90ec77d51aec
I20240502 14:06:34.407912 15877 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.408048 15877 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.408461 15877 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.409055 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" is_pre_election: true
I20240502 14:06:34.409201 29159 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:06:34.409572 29358 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters:
W20240502 14:06:34.409665 29357 leader_election.cc:334] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.409814 15877 raft_consensus.cc:2781] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:06:34.409873 15877 raft_consensus.cc:482] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting leader election (detected failure of leader 12348ca540074b32912fdd6333484e0b)
I20240502 14:06:34.409914 15877 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:34.411000 15877 raft_consensus.cc:504] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.411295 15877 leader_election.cc:288] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Requested vote from peers bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.411525 29159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a476d1664de7456ab21ddbf2e12b4f0d" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86"
I20240502 14:06:34.411621 29159 raft_consensus.cc:3037] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:06:34.412313 29357 leader_election.cc:334] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.412516 29159 raft_consensus.cc:2454] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 3.
I20240502 14:06:34.412791 29358 leader_election.cc:302] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd40abfe1fbd48529f9ebbcc4d3abf86, edce308627b14fa6968d90ec77d51aec; no voters: 12348ca540074b32912fdd6333484e0b
I20240502 14:06:34.412937 15877 raft_consensus.cc:2781] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:06:34.413102 15877 raft_consensus.cc:686] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [term 3 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:06:34.413226 15877 consensus_queue.cc:237] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.414881 28884 catalog_manager.cc:5484] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 2 to 3, leader changed from 12348ca540074b32912fdd6333484e0b (127.16.217.6) to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 3 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:34.444037 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a): 4840 bytes on disk
I20240502 14:06:34.444861 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(1dda4193aaaf459fb5049ff49541229a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 14:06:34.467504 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.473227 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.476171 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.476364 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.479753 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.482167 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.492524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.495576 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.503051 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.521690 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:34.602370 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet e6787832a36648aca61dd26a363089d5: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:34.619719 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.621680 15877 raft_consensus.cc:482] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:06:34.621790 15877 raft_consensus.cc:504] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.622195 15877 leader_election.cc:288] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.622639 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:34.622781 29292 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 2.
I20240502 14:06:34.623344 29359 leader_election.cc:302] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
I20240502 14:06:34.624670 15877 raft_consensus.cc:2781] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:06:34.624737 15877 raft_consensus.cc:482] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:06:34.624783 15877 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:34.625120 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.625969 15877 raft_consensus.cc:504] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.626283 15877 leader_election.cc:288] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Requested vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737)
I20240502 14:06:34.626924 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0176fb1a25a94c1c9d8cf0dc63577e91" candidate_uuid: "edce308627b14fa6968d90ec77d51aec" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5"
I20240502 14:06:34.627032 29293 raft_consensus.cc:3037] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:06:34.628103 29293 raft_consensus.cc:2454] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edce308627b14fa6968d90ec77d51aec in term 3.
I20240502 14:06:34.629621 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.629842 29359 leader_election.cc:302] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd692a8958c241fca3aec5cca3a0ded5, edce308627b14fa6968d90ec77d51aec; no voters:
W20240502 14:06:34.630714 29357 leader_election.cc:334] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.631124 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.634256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.646951 29403 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:34.652598 29357 leader_election.cc:334] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.653285 15877 raft_consensus.cc:2781] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:06:34.653391 15877 raft_consensus.cc:686] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [term 3 LEADER]: Becoming Leader. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Running, Role: LEADER
I20240502 14:06:34.653530 15877 consensus_queue.cc:237] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } }
I20240502 14:06:34.653883 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.656344 28876 catalog_manager.cc:5484] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec reported cstate change: term changed from 2 to 3, leader changed from 12348ca540074b32912fdd6333484e0b (127.16.217.6) to edce308627b14fa6968d90ec77d51aec (127.16.217.4). New cstate: current_term: 3 leader_uuid: "edce308627b14fa6968d90ec77d51aec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } } peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "12348ca540074b32912fdd6333484e0b" member_type: VOTER last_known_addr { host: "127.16.217.6" port: 38737 } } }
I20240502 14:06:34.660713 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.676044 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:34.677464 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.721324 16381 ts_tablet_manager.cc:902] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:34.721899 16381 tablet_copy_client.cc:308] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:06:34.722442 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 15513e220e144883b216c4f3ac8ade0b from peer edce308627b14fa6968d90ec77d51aec ({username='slave'} at 127.16.217.4:56987)
I20240502 14:06:34.722558 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 15513e220e144883b216c4f3ac8ade0b from peer edce308627b14fa6968d90ec77d51aec at {username='slave'} at 127.16.217.4:56987: session id = edce308627b14fa6968d90ec77d51aec-15513e220e144883b216c4f3ac8ade0b
I20240502 14:06:34.723564 29048 tablet_copy_source_session.cc:205] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:34.724351 16381 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:34.726209 16381 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.13
I20240502 14:06:34.726377 16381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:34.727293 16381 tablet_copy_client.cc:783] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:34.731362 16381 tablet_copy_client.cc:647] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:34.732637 16381 tablet_copy_client.cc:523] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:34.733702 16381 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Bootstrap starting.
W20240502 14:06:34.736845 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:34.741343 29426 raft_consensus.cc:1203] T 1dda4193aaaf459fb5049ff49541229a P edce308627b14fa6968d90ec77d51aec [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 14:06:34.744772 16381 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:34.744982 16381 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Bootstrap complete.
I20240502 14:06:34.745118 16381 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.003s
I20240502 14:06:34.745411 16381 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:34.745498 16381 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: edce308627b14fa6968d90ec77d51aec, State: Initialized, Role: FOLLOWER
I20240502 14:06:34.745586 16381 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "edce308627b14fa6968d90ec77d51aec" member_type: VOTER last_known_addr { host: "127.16.217.4" port: 45347 } }
I20240502 14:06:34.745776 16381 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:06:34.746160 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session edce308627b14fa6968d90ec77d51aec-15513e220e144883b216c4f3ac8ade0b received from {username='slave'} at 127.16.217.4:56987
I20240502 14:06:34.746229 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session edce308627b14fa6968d90ec77d51aec-15513e220e144883b216c4f3ac8ade0b on tablet 15513e220e144883b216c4f3ac8ade0b with peer edce308627b14fa6968d90ec77d51aec
I20240502 14:06:34.751137 29292 raft_consensus.cc:1261] T d5ffcf19e66e4adca28d87cbe2245964 P cd692a8958c241fca3aec5cca3a0ded5 [term 3 FOLLOWER]: Refusing update from remote peer c59dc7abff9749abab221ce409d49f59: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:06:34.751530 15976 consensus_queue.cc:1035] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 14:06:34.754880 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:34.805385 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.805387 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.808841 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:34.818228 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:06:34.819787 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:34.827123 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:34.859234 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:06:34.869287 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 14:06:34.931447 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:34.937438 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:34.940259 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:34.945585 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:34.947889 29477 maintenance_manager.cc:419] P edce308627b14fa6968d90ec77d51aec: Scheduling UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b): 4586 bytes on disk
I20240502 14:06:34.948129 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:34.948639 29361 maintenance_manager.cc:643] P edce308627b14fa6968d90ec77d51aec: UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 14:06:34.949898 29160 raft_consensus.cc:1261] T a476d1664de7456ab21ddbf2e12b4f0d P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 14:06:34.950289 15877 consensus_queue.cc:1035] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:06:35.012521 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.016165 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.019989 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.029119 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.032608 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.062152 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.062817 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.063025 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.064175 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:35.066375 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:35.072589 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.100486 29292 raft_consensus.cc:1261] T 0176fb1a25a94c1c9d8cf0dc63577e91 P cd692a8958c241fca3aec5cca3a0ded5 [term 3 FOLLOWER]: Refusing update from remote peer edce308627b14fa6968d90ec77d51aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 14:06:35.100950 16436 consensus_queue.cc:1035] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:06:35.107039 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:35.149840 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.150588 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.150641 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.152745 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.155589 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:35.193166 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:35.228096 29426 raft_consensus.cc:1203] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.14->[3.15-3.15] Dedup: 3.15->[]
W20240502 14:06:35.242657 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:35.247200 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.247517 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.250762 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.253233 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.261023 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.338001 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.342197 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.342355 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.343219 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.365408 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.421392 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.423377 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.423663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.427590 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:35.436323 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:35.436654 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:35.466256 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20240502 14:06:35.482803 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } }
I20240502 14:06:35.484097 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20240502 14:06:35.484982 16841 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 14:06:35.494541 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.495549 16841 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:35.496858 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.499861 28876 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 15 to 16, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:35.500419 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER edce308627b14fa6968d90ec77d51aec (127.16.217.4) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } }
I20240502 14:06:35.502390 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.503572 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.504251 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.517910 29403 tablet_service.cc:1514] Processing DeleteTablet for tablet 15513e220e144883b216c4f3ac8ade0b with delete_type TABLET_DATA_TOMBSTONED (TS edce308627b14fa6968d90ec77d51aec not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:45210
I20240502 14:06:35.518280 16986 tablet_replica.cc:330] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: stopping tablet replica
I20240502 14:06:35.518193 29027 consensus_queue.cc:237] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:35.518409 16986 raft_consensus.cc:2229] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 14:06:35.518550 16986 raft_consensus.cc:2258] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 14:06:35.519150 16986 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:06:35.519670 31178 raft_consensus.cc:1261] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Refusing update from remote peer a7221b90d0aa4770a8ce5522103a96f4: Log matching property violated. Preceding OpId in replica: term: 3 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240502 14:06:35.520100 16110 consensus_queue.cc:1035] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240502 14:06:35.521296 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:06:35.521548 28961 consensus_peers.cc:485] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 -> Peer bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2:35841): Couldn't send request to peer bd40abfe1fbd48529f9ebbcc4d3abf86. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:06:35.521798 16110 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 [term 3 LEADER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:35.522246 16986 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20240502 14:06:35.522320 16986 log.cc:1196] T 15513e220e144883b216c4f3ac8ade0b P edce308627b14fa6968d90ec77d51aec: Deleting WAL directory at /tmp/dist-test-taskhNHehp/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714658377875918-17252-0/raft_consensus-itest-cluster/ts-3/wal/wals/15513e220e144883b216c4f3ac8ade0b
I20240502 14:06:35.522221 31178 raft_consensus.cc:2932] T 15513e220e144883b216c4f3ac8ade0b P c59dc7abff9749abab221ce409d49f59 [term 3 FOLLOWER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
I20240502 14:06:35.523214 28870 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 15513e220e144883b216c4f3ac8ade0b with cas_config_opid_index 16: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 14:06:35.523504 28868 catalog_manager.cc:4830] TS edce308627b14fa6968d90ec77d51aec (127.16.217.4:45347): tablet 15513e220e144883b216c4f3ac8ade0b (table rebalance_test_table_9 [id=4d80a42354b345198c24d894e6eed687]) successfully deleted
I20240502 14:06:35.524984 28883 catalog_manager.cc:5484] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4 reported cstate change: config changed from index 16 to 17, VOTER bd40abfe1fbd48529f9ebbcc4d3abf86 (127.16.217.2) added. New cstate: current_term: 3 leader_uuid: "a7221b90d0aa4770a8ce5522103a96f4" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } } }
W20240502 14:06:35.536983 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 184 similar messages]
W20240502 14:06:35.550487 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:35.597435 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.597713 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.604539 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.605312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.606019 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:35.675299 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:06:35.679778 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 14:06:35.681270 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:35.682971 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:35.694249 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.695684 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.698617 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.699165 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.700335 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:35.768201 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:06:35.808362 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:35.813275 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.815317 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.818477 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.825565 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.825829 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.827522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.840713 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.841250 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.867741 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.892199 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.895155 31235 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskhNHehp/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 14:01:19.823642 31239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:01:19.823899 31239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:01:19.834275 31239 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:01:20.768194 31239 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:01:21.257828 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.257903 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.257952 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.258016 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.258054 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.258090 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.258126 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.258164 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.258189 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.258221 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.258247 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:21.258271 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:21.258296 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:21.258320 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 22dbbf207ba0440392e71a10c2ca78ca
I20240502 14:01:21.329690 31239 rebalancer_tool.cc:1074] tablet a75497cf22014c0cb9d079b448284b3d: 'edce308627b14fa6968d90ec77d51aec' -> '12348ca540074b32912fdd6333484e0b' move scheduled
I20240502 14:01:21.406105 31239 rebalancer_tool.cc:1074] tablet ff65ce0568514e89894ec845fbda31f3: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '33ebf308f6f24e6e8768ef5f14ddc785' move scheduled
I20240502 14:01:22.400976 31239 rebalancer_tool.cc:1074] tablet 5e1b37b67e104154b6755c1762f8f6ff: 'a7221b90d0aa4770a8ce5522103a96f4' -> 'c59dc7abff9749abab221ce409d49f59' move scheduled
I20240502 14:01:22.412662 31239 rebalancer_tool.cc:1084] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '22dbbf207ba0440392e71a10c2ca78ca' move ignored: Illegal state: tablet 015b587864d54da8aee36c3afd968e86: replica 22dbbf207ba0440392e71a10c2ca78ca is already present
I20240502 14:01:22.412770 31239 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:22.843117 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.843227 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.843310 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:22.843366 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.843407 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:22.843443 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:22.843479 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 3830162daeef4e3c9729520f72d4b634: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.843516 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.843552 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:22.843601 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:22.843636 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.843673 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 4d80a42354b345198c24d894e6eed687: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:22.843709 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server edce308627b14fa6968d90ec77d51aec to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:22.843748 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:22.843784 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 103008b3b2e64508ae956ee1a3e79272: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server 12348ca540074b32912fdd6333484e0b
I20240502 14:01:22.884465 31239 rebalancer_tool.cc:1074] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:23.015336 31239 rebalancer_tool.cc:1074] tablet 372e9cc5aaed4b80b3b451efdb3e94fc: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move scheduled
I20240502 14:01:23.113711 31239 rebalancer_tool.cc:1074] tablet f05a1a0bf1314d098f2bf65df58222e2: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:23.134666 31239 rebalancer_tool.cc:1084] tablet 4aa944cd0b82429da02bd3dd35e0cce7: 'edce308627b14fa6968d90ec77d51aec' -> '12348ca540074b32912fdd6333484e0b' move ignored: Illegal state: tablet 4aa944cd0b82429da02bd3dd35e0cce7: replica 12348ca540074b32912fdd6333484e0b is already present
I20240502 14:01:23.134747 31239 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:01:23.487133 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server c59dc7abff9749abab221ce409d49f59
W20240502 14:01:23.487313 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 752701223a054996b6bce8d893df0637: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 12348ca540074b32912fdd6333484e0b
W20240502 14:01:23.487388 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server cd692a8958c241fca3aec5cca3a0ded5
W20240502 14:01:23.487432 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 15e53fa6a89946a5996190ce29b908ce: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:23.487473 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6e061e16779c475bbfadb37b5432fc9f: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.487543 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c6db7e4785cf4c7880889eda909495f9: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.487607 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 13d99761e8c04c928e4c803a94305c14: could not find any suitable replica to move from server bd40abfe1fbd48529f9ebbcc4d3abf86 to server 22dbbf207ba0440392e71a10c2ca78ca
W20240502 14:01:23.487645 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 79af070d92f044a5b8d88e2cf5578abd: could not find any suitable replica to move from server cd692a8958c241fca3aec5cca3a0ded5 to server 33ebf308f6f24e6e8768ef5f14ddc785
W20240502 14:01:23.487740 31239 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a44a326932a24e619cf3633c21b3bab0: could not find any suitable replica to move from server a7221b90d0aa4770a8ce5522103a96f4 to server c59dc7abff9749abab221ce409d49f59
I20240502 14:01:23.537894 31239 rebalancer_tool.cc:1074] tablet 400da91cbe234c79859b23ebe1643eb0: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:23.577009 31239 rebalancer_tool.cc:1074] tablet c88f85f143d845a1b7066244c606f8ba: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:23.588454 31239 rebalancer_tool.cc:1074] tablet ed713f89a4fd47d79f03711dd14c1334: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:25.465018 31239 rebalancer_tool.cc:1136] tablet ed713f89a4fd47d79f03711dd14c1334: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move completed: OK
I20240502 14:01:26.020537 31239 rebalancer_tool.cc:1136] tablet 5e1b37b67e104154b6755c1762f8f6ff: 'a7221b90d0aa4770a8ce5522103a96f4' -> 'c59dc7abff9749abab221ce409d49f59' move completed: OK
I20240502 14:01:26.985155 31239 rebalancer_tool.cc:1136] tablet f05a1a0bf1314d098f2bf65df58222e2: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move completed: OK
I20240502 14:01:26.998359 31239 rebalancer_tool.cc:1074] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move scheduled
I20240502 14:01:27.013584 31239 rebalancer_tool.cc:1074] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:27.096452 31239 rebalancer_tool.cc:1136] tablet 0176fb1a25a94c1c9d8cf0dc63577e91: 'a7221b90d0aa4770a8ce5522103a96f4' -> '' move completed: OK
I20240502 14:01:27.672684 31239 rebalancer_tool.cc:1136] tablet c88f85f143d845a1b7066244c606f8ba: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:28.215790 31239 rebalancer_tool.cc:1136] tablet ff65ce0568514e89894ec845fbda31f3: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '33ebf308f6f24e6e8768ef5f14ddc785' move completed: OK
I20240502 14:01:28.245097 31239 rebalancer_tool.cc:1136] tablet a75497cf22014c0cb9d079b448284b3d: 'edce308627b14fa6968d90ec77d51aec' -> '12348ca540074b32912fdd6333484e0b' move completed: OK
I20240502 14:01:28.273880 31239 rebalancer_tool.cc:1136] tablet 785828dee4594ca8813b9418cc5dba2d: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:28.322337 31239 rebalancer_tool.cc:1136] tablet 372e9cc5aaed4b80b3b451efdb3e94fc: 'edce308627b14fa6968d90ec77d51aec' -> '22dbbf207ba0440392e71a10c2ca78ca' move completed: OK
I20240502 14:01:28.335186 31239 rebalancer_tool.cc:1074] tablet 19784717201a464c84607d2a43e651c2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move scheduled
I20240502 14:01:28.448326 31239 rebalancer_tool.cc:1074] tablet cfd44eeeec934413a5001673430e6500: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:28.481084 31239 rebalancer_tool.cc:1074] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:29.144328 31239 rebalancer_tool.cc:1136] tablet cfd44eeeec934413a5001673430e6500: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:29.642743 31239 rebalancer_tool.cc:1136] tablet dcc536794ffb453495d025f29ea152bc: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:29.670214 31239 rebalancer_tool.cc:1136] tablet 400da91cbe234c79859b23ebe1643eb0: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:29.697734 31239 rebalancer_tool.cc:1074] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move scheduled
I20240502 14:01:29.733043 31239 rebalancer_tool.cc:1074] tablet 2b1adebdbd104796b927963e86a15295: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:29.751546 31239 rebalancer_tool.cc:1074] tablet 81b2d5d0f4044106bed079c3faa72447: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:30.251217 31239 rebalancer_tool.cc:1136] tablet 015b587864d54da8aee36c3afd968e86: 'cd692a8958c241fca3aec5cca3a0ded5' -> '' move completed: OK
I20240502 14:01:30.260605 31239 rebalancer_tool.cc:1136] tablet 86cfc7d5709f414aac098ae3c05a48a3: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:30.279675 31239 rebalancer_tool.cc:1136] tablet 81b2d5d0f4044106bed079c3faa72447: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
I20240502 14:01:30.290689 31239 rebalancer_tool.cc:1136] tablet 19784717201a464c84607d2a43e651c2: 'bd40abfe1fbd48529f9ebbcc4d3abf86' -> '' move completed: OK
I20240502 14:01:30.299221 31239 rebalancer_tool.cc:1074] tablet 15513e220e144883b216c4f3ac8ade0b: 'edce308627b14fa6968d90ec77d51aec' -> '' move scheduled
I20240502 14:01:31.323895 31239 rebalancer_tool.cc:1136] tablet 2b1adebdbd104796b927963e86a15295: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
W20240502 14:06:33.296156 15534 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:06:33.051025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.217.6:38737 (local address 127.0.0.1:40804)
0502 14:06:33.051251 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0502 14:06:33.051264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 14:06:33.051323 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:06:33.295360 (+244037us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157}
I20240502 14:06:35.722664 31239 rebalancer_tool.cc:1136] tablet 15513e220e144883b216c4f3ac8ade0b: 'edce308627b14fa6968d90ec77d51aec' -> '' move completed: OK
Illegal state: tablet server 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): unacceptable health status UNAVAILABLE
W20240502 14:06:35.930012 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:35.951826 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:35.953570 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:35.953722 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:35.961460 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:35.970544 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:35.982832 17195 ts_tablet_manager.cc:902] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Initiating tablet copy from peer a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835)
I20240502 14:06:35.983372 17195 tablet_copy_client.cc:308] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Beginning tablet copy session from remote peer at address 127.16.217.1:37835
I20240502 14:06:35.983777 29048 tablet_copy_service.cc:140] P a7221b90d0aa4770a8ce5522103a96f4: Received BeginTabletCopySession request for tablet 15513e220e144883b216c4f3ac8ade0b from peer bd40abfe1fbd48529f9ebbcc4d3abf86 ({username='slave'} at 127.16.217.2:35737)
I20240502 14:06:35.983879 29048 tablet_copy_service.cc:161] P a7221b90d0aa4770a8ce5522103a96f4: Beginning new tablet copy session on tablet 15513e220e144883b216c4f3ac8ade0b from peer bd40abfe1fbd48529f9ebbcc4d3abf86 at {username='slave'} at 127.16.217.2:35737: session id = bd40abfe1fbd48529f9ebbcc4d3abf86-15513e220e144883b216c4f3ac8ade0b
I20240502 14:06:35.984879 29048 tablet_copy_source_session.cc:205] T 15513e220e144883b216c4f3ac8ade0b P a7221b90d0aa4770a8ce5522103a96f4: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 14:06:35.985603 17195 ts_tablet_manager.cc:1875] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 14:06:35.987982 17195 ts_tablet_manager.cc:1888] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 14:06:35.988166 17195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15513e220e144883b216c4f3ac8ade0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:06:35.989337 17195 tablet_copy_client.cc:783] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 5 data blocks...
I20240502 14:06:35.994027 17195 tablet_copy_client.cc:647] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Starting download of 1 WAL segments...
I20240502 14:06:35.995124 17195 tablet_copy_client.cc:523] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:06:35.996311 17195 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap starting.
I20240502 14:06:36.005046 17195 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:06:36.005293 17195 tablet_bootstrap.cc:492] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Bootstrap complete.
I20240502 14:06:36.005438 17195 ts_tablet_manager.cc:1367] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20240502 14:06:36.005769 17195 raft_consensus.cc:348] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:36.005860 17195 raft_consensus.cc:729] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd40abfe1fbd48529f9ebbcc4d3abf86, State: Initialized, Role: FOLLOWER
I20240502 14:06:36.005977 17195 consensus_queue.cc:260] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 3.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c59dc7abff9749abab221ce409d49f59" member_type: VOTER last_known_addr { host: "127.16.217.8" port: 38573 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "bd40abfe1fbd48529f9ebbcc4d3abf86" member_type: VOTER last_known_addr { host: "127.16.217.2" port: 35841 } }
I20240502 14:06:36.006239 17195 ts_tablet_manager.cc:1398] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:06:36.006672 29048 tablet_copy_service.cc:342] P a7221b90d0aa4770a8ce5522103a96f4: Request end of tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-15513e220e144883b216c4f3ac8ade0b received from {username='slave'} at 127.16.217.2:35737
I20240502 14:06:36.006750 29048 tablet_copy_service.cc:433] P a7221b90d0aa4770a8ce5522103a96f4: ending tablet copy session bd40abfe1fbd48529f9ebbcc4d3abf86-15513e220e144883b216c4f3ac8ade0b on tablet 15513e220e144883b216c4f3ac8ade0b with peer bd40abfe1fbd48529f9ebbcc4d3abf86
W20240502 14:06:36.032055 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:36.048350 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.048426 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.050392 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.051195 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.056843 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.137035 29208 maintenance_manager.cc:419] P bd40abfe1fbd48529f9ebbcc4d3abf86: Scheduling UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b): 4586 bytes on disk
I20240502 14:06:36.137528 29097 maintenance_manager.cc:643] P bd40abfe1fbd48529f9ebbcc4d3abf86: UndoDeltaBlockGCOp(15513e220e144883b216c4f3ac8ade0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240502 14:06:36.176728 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:06:36.178471 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.180765 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.181154 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.183794 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.183993 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:36.202674 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:36.209931 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:36.275139 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.276710 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.277940 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.278824 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.285544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.370563 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.370944 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.371454 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.374184 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.374925 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.461645 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.462105 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.462909 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.464138 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.472903 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:36.519816 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:06:36.529433 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 14:06:36.552928 29160 raft_consensus.cc:1203] T 15513e220e144883b216c4f3ac8ade0b P bd40abfe1fbd48529f9ebbcc4d3abf86 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.16->[3.17-3.17] Dedup: 3.17->[]
I20240502 14:06:36.573607 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.575242 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.576485 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.579099 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.588750 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.665277 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.667707 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.668398 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.669835 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.670635 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:36.686239 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:36.755913 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.759667 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.759907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.762171 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.767341 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.856616 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.858969 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.860370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.860535 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.869426 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:36.969542 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:36.970012 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:36.970764 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:36.981756 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:36.982913 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.065280 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.065771 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.068363 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.068563 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.076897 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:37.159821 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:37.161330 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.162928 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.163473 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.164628 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.172686 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:37.219221 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:37.251526 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.251688 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.252965 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:37.253124 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:06:37.253968 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.263402 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:37.299800 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:06:37.368228 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.368348 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.368875 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.372002 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.378587 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.461805 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.463713 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.464448 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.474226 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.474516 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:37.512238 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 27 similar messages]
W20240502 14:06:37.512890 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:06:37.541422 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:37.549831 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.550500 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.552410 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.552551 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.569499 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:37.597149 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:37.646744 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.647552 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.648073 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.651418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.669366 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.762027 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.768716 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:37.778057 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.778972 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.788908 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:37.789889 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:37.867046 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.869165 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.869441 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.871084 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.875237 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:37.882773 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:37.962527 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:37.963439 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:37.964262 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:37.967206 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:37.972396 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:37.992105 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:06:38.002928 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.050935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.054673 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.056437 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.058264 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:38.065838 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.072954 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:38.108980 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:06:38.109509 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:06:38.130779 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:38.173666 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.175010 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.176940 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.180423 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.185307 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:38.200007 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.266141 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.268693 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.269222 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:38.272565 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.280642 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.285905 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.355118 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.360877 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.369906 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.370556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.371459 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.456884 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.456974 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.457358 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.457787 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.459275 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:38.487789 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.570116 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.572954 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.572984 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.576941 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.584424 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:38.588876 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:38.662703 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.663354 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.668920 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.673128 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:38.677404 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:06:38.677994 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:38.709524 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:38.754462 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.756568 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.757158 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.757292 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.762588 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.775206 18239 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:38.775333 18239 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:38.775799 18239 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:38.776571 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:38.776733 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:06:38.778079 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:38.778177 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:38.778228 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:38.778577 18239 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:38.858143 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:38.858333 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.859114 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.863950 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.879681 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:38.885717 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 14:06:38.963080 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:38.966603 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:38.968426 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:38.970013 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:38.971135 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:38.977288 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:38.996488 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a75497cf22014c0cb9d079b448284b3d: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:39.056540 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.058212 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.061275 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.062220 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.068437 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:39.115888 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:06:39.159899 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.160439 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.160465 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.167804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.168443 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.246767 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.250739 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.252575 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.257786 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.263751 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.357980 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.361340 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.363190 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.368439 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.368938 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:39.401047 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:39.453104 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.453469 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.455474 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.456243 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.461133 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:39.472534 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 14:06:39.490595 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:06:39.545624 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.547992 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.551054 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.551087 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.556437 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.643981 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.645927 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.647090 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.654218 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.659688 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:39.664412 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 14:06:39.750735 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.751332 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.751384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:39.752379 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.760442 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.840235 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.844079 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.845122 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.847945 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.862875 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:39.885227 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 14:06:39.897426 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:06:39.915444 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:39.970695 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:39.970841 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:39.974311 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:39.974507 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:39.976838 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.064594 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.065023 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:40.066597 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.067340 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.067617 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.154026 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.154392 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.155050 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.156757 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.166380 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:40.184638 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.243331 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.243602 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.244783 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.245469 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:40.246604 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.248951 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:40.253973 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 14:06:40.301374 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c88f85f143d845a1b7066244c606f8ba: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:40.344095 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.368450 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.369379 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.372984 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.375232 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.383800 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:40.457998 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.458051 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:40.462126 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.463132 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.464795 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.465131 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:40.489526 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:06:40.509222 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:40.516368 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:40.553537 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.558643 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.559350 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:40.563776 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.578656 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.654803 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.656345 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:40.658339 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.659168 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.659633 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.660184 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:40.678136 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 26 similar messages]
W20240502 14:06:40.708518 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:06:40.713886 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.763064 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.763070 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.763397 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.766053 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:40.780516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:40.803187 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:06:40.841876 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:40.861204 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:40.864502 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.865644 18798 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:40.865768 18798 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:40.866217 18798 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:40.866888 29292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:06:40.867031 29292 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
I20240502 14:06:40.867110 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.867573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.868192 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:40.877250 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:40.877385 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:40.877424 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:40.877635 18798 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:06:40.922183 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:06:40.922823 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:40.960177 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 14:06:40.964368 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:40.966192 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:40.967716 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:40.968077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:40.979070 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.055169 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.055585 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.058698 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.070550 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.071238 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:41.090474 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:06:41.174116 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.174548 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.175891 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.191679 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.191957 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:41.235658 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:41.266268 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.268083 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.269119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.272451 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.279211 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:41.300093 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:41.356232 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.361950 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.362195 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.366014 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.376570 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.453946 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.456033 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.457136 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.458746 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.461284 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.567268 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.568444 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.572027 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.573822 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.576426 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:41.649247 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:06:41.662731 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.663393 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.665187 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.665272 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.673480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:41.681763 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:41.757397 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.758877 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.759549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.763880 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.764123 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.854283 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.857275 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.858403 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:41.861462 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.868713 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.962487 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:41.964821 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:41.964857 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:41.966964 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:41.967237 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:41.994382 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 14:06:42.008249 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:42.054293 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.055160 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.059654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.060351 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.060974 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.161401 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.163838 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.164301 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.164834 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.167054 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:42.186336 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 14:06:42.205665 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:42.248163 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.248649 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.249208 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.253082 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.257091 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:42.306536 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:42.351313 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.356086 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.357753 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.358242 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.360572 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.449008 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.451524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.452453 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.452504 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.454958 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:42.534050 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:42.548709 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.553592 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.555668 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.558267 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.577350 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:42.585141 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:06:42.650063 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.652495 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.654650 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.664691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.665664 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:42.686208 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:42.750708 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.752561 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:42.753899 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:42.754675 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.757977 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.764580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.849567 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.853322 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.858708 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.860167 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:42.871541 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:42.876730 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:06:42.899480 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:06:42.939534 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:06:42.948493 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 14:06:42.960558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:42.962440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:42.962672 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:42.965451 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:42.973454 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.041756 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:06:43.041898 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:43.058789 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.059432 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.059477 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.060142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.060716 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.144474 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:43.160482 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.160482 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.160832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.161332 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.162717 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.186128 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:43.250033 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.250573 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.251267 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.253907 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.257360 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.263729 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:43.349967 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.355682 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.356177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.367846 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.379271 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.385450 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:06:43.442047 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240502 14:06:43.451205 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.451938 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.452010 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.453598 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:43.462420 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.547856 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.549336 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.549634 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.553608 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.577904 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:43.595084 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:43.652222 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.652292 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.656291 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.659634 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:43.661273 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:43.694432 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:43.705082 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:06:43.750293 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.750555 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.751740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.761451 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:43.764889 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.854822 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.855906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:43.856086 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.867331 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.868326 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:43.956328 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:43.958474 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:43.960453 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:43.961485 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:43.978595 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.054119 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.054410 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.056558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.063112 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.064610 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.159886 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.163524 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.164435 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.166630 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.175460 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:44.175673 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:44.212172 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 14:06:44.253010 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.253991 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.254223 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.255024 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.271654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.352602 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.361807 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.363184 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.363813 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.379217 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:44.431735 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240502 14:06:44.452621 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.453807 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.454699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.455838 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.460445 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.548905 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.550205 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.550802 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.553274 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.566390 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:44.596541 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:06:44.612437 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:44.631734 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:44.638880 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.644124 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.644405 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.646505 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.657687 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:44.677723 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 14:06:44.746790 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.746925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:44.748104 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.748524 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.748567 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.837280 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.839843 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.840155 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:44.841656 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:44.843070 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:44.843556 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.940685 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:44.942906 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:44.949549 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:44.954778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:44.955359 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:06:44.965492 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.046198 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.047488 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.047663 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.058569 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.060554 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.139668 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.141041 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.143864 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.146875 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.156177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:45.170852 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:45.233278 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.233327 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.233788 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.236434 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.242774 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:45.277864 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 14:06:45.293784 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:45.345840 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.346372 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.349309 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.349807 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.353596 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:45.377305 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:45.435473 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.437255 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.441531 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.442775 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.452760 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:45.489702 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:06:45.528841 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:45.529690 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.535046 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.535728 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.542372 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.543267 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:45.584838 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:45.628046 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.628515 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.630175 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:45.632516 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:45.633052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.633384 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.732719 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.732877 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.735062 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.736135 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:45.755504 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:45.755893 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:45.777204 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:06:45.777521 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:06:45.829773 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.832899 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:45.839843 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.840142 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.856581 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:45.868700 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240502 14:06:45.889122 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240502 14:06:45.938112 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:45.939623 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:45.940153 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:45.941352 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:45.942715 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:45.944654 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.033598 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.034559 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.034714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.035024 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.036722 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:46.053056 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:46.136606 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.141345 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.144467 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.145260 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.149991 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:46.185111 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:06:46.223480 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.226625 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.227015 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.228359 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.232452 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:46.248358 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:46.298290 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:46.327606 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.327630 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.329665 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.337425 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.344699 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.419395 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.419445 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.422341 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.428658 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.435846 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.521669 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.522338 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.522387 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.524538 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.535651 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.615602 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.615772 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.616295 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.618769 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.621462 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:46.696439 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:46.720748 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.722546 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.723663 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.735594 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.736311 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.815539 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.815912 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.820559 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.821367 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:46.825286 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:46.856973 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 14:06:46.915097 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:46.918102 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:46.918880 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:46.919441 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:46.931865 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.020344 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.022202 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.023277 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.023617 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.023617 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:47.046558 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:06:47.118429 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.120062 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.120388 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.120617 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.123775 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:47.175509 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:47.207715 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.210139 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.212697 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.215667 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.217886 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:47.220613 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:06:47.235949 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240502 14:06:47.236805 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a91fd318f0844c1961f16823a7ccc88: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:47.315341 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.317653 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.319535 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.325443 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.332679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.415194 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.415436 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.415740 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.417683 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.418282 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:47.432142 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 14:06:47.485251 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:47.510767 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.513024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.515509 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.515743 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.517413 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:47.585937 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:47.609459 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.618384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.621075 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.623557 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.624624 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.711585 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.711694 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.713415 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.714241 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.714716 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:47.723915 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:47.774142 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:06:47.779080 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:47.814065 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:06:47.814754 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.815773 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.816332 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.821069 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:47.832712 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:47.861564 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:06:47.864799 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 14:06:47.918699 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:47.920817 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:47.921451 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:47.924419 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:47.933445 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:47.939894 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:06:47.943331 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240502 14:06:48.012420 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.015098 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.017284 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.017946 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.023329 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:48.116101 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:48.117245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.118956 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.120317 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.121421 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.122489 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.211856 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.212865 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.213389 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.214182 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.214553 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:48.263917 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 14:06:48.290108 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:48.308336 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:48.313025 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.313400 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.313860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.315693 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:48.324842 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 14:06:48.342623 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.411551 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.411633 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.418751 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.423815 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.424954 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:48.487108 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:48.512413 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.513036 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.513413 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.514711 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.516993 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:48.533427 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c88f85f143d845a1b7066244c606f8ba: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:48.606216 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.607453 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.608958 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.609617 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.616483 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.710779 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.711035 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.711226 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.720892 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.729355 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:48.795642 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:48.808822 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.809432 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.810361 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.812305 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:48.818763 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:48.820459 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:48.883965 22110 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:48.884079 22110 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:48.884557 22110 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:48.885107 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:06:48.885242 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:06:48.886394 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:48.886505 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:48.886544 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:48.886682 22110 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:48.923398 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:48.926923 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:48.933071 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:48.933422 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:48.938040 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.024004 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.025441 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.027558 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.027730 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.028862 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.128127 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.128679 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.130020 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.133795 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.136433 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:49.208514 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
I20240502 14:06:49.220818 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.221822 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.224861 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.225033 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.225471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:49.263265 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:49.326925 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.329353 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.331175 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.332964 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.335429 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:49.387195 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:49.418994 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.423547 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.424093 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.428050 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.428463 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.523166 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.524899 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.526648 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.534078 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.534955 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.621125 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.621510 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.622618 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.623481 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.625777 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:49.689043 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:49.722147 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.724565 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.725982 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.731675 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.740619 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:49.760668 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 14:06:49.798188 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:49.820523 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.821025 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:49.821892 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.823122 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.832098 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.918639 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:49.923403 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:49.923764 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:49.924880 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:49.928944 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:49.967114 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240502 14:06:50.036648 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:50.038364 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:50.040093 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.043475 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.044157 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:50.056522 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:50.056767 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:50.081156 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:50.097704 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:50.128983 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.133610 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.134299 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.135460 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.155390 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.224735 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.228740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:50.229833 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:50.230381 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.233050 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.239063 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:50.244705 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:06:50.337810 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.339527 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.343187 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:50.349306 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:50.354108 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:50.356493 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:50.361060 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:50.373870 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240502 14:06:50.423751 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:50.428211 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 14:06:50.440721 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.440760 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.440991 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:50.441857 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:50.442723 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.444149 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.530453 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.534123 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.534332 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.535760 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.540652 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.626296 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.627229 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.629078 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.630076 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.631294 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.730195 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.732652 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.733496 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.737457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.744828 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:50.764160 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:50.828727 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.829553 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:50.831678 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.831833 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.853498 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:50.881278 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:50.882827 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:50.934044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:50.934748 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:50.936259 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:50.936854 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:50.942643 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:51.001180 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:06:51.028398 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.028843 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.032599 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.033286 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.036430 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:51.078349 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:51.086761 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240502 14:06:51.125720 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:51.139737 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.140251 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.141115 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.142611 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.143961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.224258 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.225577 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.225593 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.232563 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.233791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.318790 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.321249 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.322288 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.332149 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.340687 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:51.357652 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:06:51.358240 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:51.419570 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.421134 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.425853 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.432574 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.432727 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.522440 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.523767 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.525064 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.526646 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.527338 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.613611 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.622207 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.622725 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.624526 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.624539 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.715289 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.715919 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.718562 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.718652 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.727463 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:51.734359 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:51.761399 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 14:06:51.809389 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:51.811061 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.811529 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.814065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.824616 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:51.860736 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:51.923045 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:51.925290 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:51.925793 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:51.938472 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:51.941032 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.020002 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.020251 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.021643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.023757 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.023770 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.117317 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.118889 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.120119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.121205 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.121291 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:52.193897 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:52.205000 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.207307 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.207782 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.208036 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.213403 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:52.258405 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:52.313823 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.318475 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.319078 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.319885 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.326275 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.399864 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.402848 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.406359 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.408324 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.408701 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.501976 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.502801 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.504698 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.505748 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.525008 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:52.564188 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 14:06:52.588996 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:52.599725 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.601089 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.601688 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.601970 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.603130 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:52.667018 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:52.667881 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:52.694931 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:52.696389 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240502 14:06:52.697793 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.697815 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.698604 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.712363 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:52.719314 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:52.783867 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:52.798240 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.798274 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.802403 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:52.804010 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.811318 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:52.833645 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:52.900377 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:52.903249 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:52.910213 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:52.911562 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:52.911602 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:52.946167 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:52.973405 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240502 14:06:53.004187 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.004937 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.005251 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.007957 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.026180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.104790 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.106637 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.109871 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.110539 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.112663 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:53.198150 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 14:06:53.199105 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.200124 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.200523 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.201077 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.217350 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:53.306979 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:53.307957 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.308161 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.308492 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.312222 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.319031 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:53.338137 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:53.410707 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.411283 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.412092 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.412876 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.413568 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:53.431088 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 14:06:53.435619 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:53.472107 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:53.500025 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.501798 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.503604 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.504940 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.508419 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:53.521631 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:53.594864 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.596740 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.597951 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.598161 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.604432 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.715293 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.717448 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.717514 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.718235 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.729859 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:53.738087 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:53.812991 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.814522 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.816401 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.816699 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:53.825903 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:53.877008 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:53.910440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:53.913304 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:53.913691 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:53.914980 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:53.922374 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.002733 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.003875 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.004631 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.005035 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.012396 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.112517 24448 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:06:54.112643 24448 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:06:54.113077 24448 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:06:54.113332 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:06:54.113483 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:06:54.114166 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240502 14:06:54.116034 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:06:54.116258 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:06:54.116341 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:06:54.116568 24448 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:06:54.118206 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.123005 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.123504 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.129467 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.134312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.211537 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.213187 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.213892 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.215447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.216531 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:54.296319 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:54.302981 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.304586 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.305241 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.306244 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.306820 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:54.332404 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:06:54.394879 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.394896 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.395498 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.396425 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.398411 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.508903 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.509097 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.512421 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.513492 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.525950 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:54.590138 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:54.606578 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.607141 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.607621 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.609251 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.625732 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.694908 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.698324 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.699056 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.705447 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.710115 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:54.765190 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:54.791263 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.792227 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.792418 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:54.792412 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.792804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:54.796329 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 14:06:54.911526 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:54.911880 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:54.915607 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:54.916710 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:54.917029 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:54.944422 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:54.964427 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240502 14:06:55.011507 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.011515 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.011876 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.013204 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.014067 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:55.082973 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:55.084893 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240502 14:06:55.109684 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.110354 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.110512 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.115203 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.126456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:55.147940 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:55.203155 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:55.204432 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.205687 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.206415 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.216894 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.218760 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:55.236568 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:55.257233 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:55.309679 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.309896 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.311810 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:55.312799 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:55.313287 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.316546 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.408123 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.411768 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.411837 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.412376 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.412488 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:55.429147 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:55.505563 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.508337 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.508337 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.508337 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.516623 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.600790 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.602990 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.603252 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.605140 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.608218 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:55.681012 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 14:06:55.706014 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.708869 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.712427 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.719615 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.727834 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:55.794473 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:55.805804 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.806062 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.806339 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:55.806919 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.816756 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:55.853747 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:55.909137 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:55.912598 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:55.913170 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:55.913313 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:55.915354 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:55.934123 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:56.010619 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:56.012369 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.014739 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.022226 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.023784 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.026594 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.103600 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.107713 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.108845 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.110761 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.113760 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:56.136737 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:56.178450 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:06:56.194005 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.197014 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.197162 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:56.200932 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:56.204583 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.206669 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:56.213383 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:56.313759 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.314142 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.316664 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.316956 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.317567 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.406823 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.408004 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.411777 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.416615 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.417997 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:56.492869 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:06:56.494139 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:56.495108 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.495541 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.501116 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.506580 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.508208 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:56.560321 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:56.591681 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.591676 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.592011 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.594394 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.608757 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.708746 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.709250 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.710577 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.714054 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.721362 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.799731 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.803107 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:56.803988 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.808909 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.808949 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.899834 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.904348 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:56.905165 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:56.913115 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.917547 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:56.970245 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:06:56.997495 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:56.997869 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:56.998162 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.002622 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.012961 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:57.082636 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:57.106786 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.110445 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.111325 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.113443 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.114120 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.200048 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.204061 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.204509 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.204965 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.212101 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.293681 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.295363 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.295544 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.295554 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.305424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:57.324820 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:57.350396 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240502 14:06:57.386561 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.388255 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.392688 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.393134 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.394623 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.496837 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.497694 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.506744 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.508667 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.509064 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:57.569244 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:57.591631 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:57.592722 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:57.594084 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.595979 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.597138 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.603915 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:57.618374 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 14:06:57.648864 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:57.686810 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.687527 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.689800 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.691212 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.699469 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:57.722599 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 14:06:57.758632 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:57.780452 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:57.781258 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.787722 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.788879 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.790246 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.790338 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:57.852254 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:57.895212 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.897248 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.897917 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.900041 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:57.916404 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:57.994124 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:57.995018 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:57.998167 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:57.998416 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.010813 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.094877 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.095402 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.097976 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.098668 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.103560 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:58.117986 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:06:58.118641 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 14:06:58.186313 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.187427 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.188155 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.189687 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.193238 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:58.211949 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:06:58.267414 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:58.289713 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.289923 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.290390 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.292542 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.301374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:58.333516 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:58.377185 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.381387 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.390004 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.391099 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.393101 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.480602 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.482580 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:58.482635 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:58.486204 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.487474 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.492410 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:06:58.553403 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:06:58.569085 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.569442 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.570394 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.577040 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.585188 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:58.653542 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:58.673955 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.674329 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.676510 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.677644 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:58.687834 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:58.690812 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.773124 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.775090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.775310 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.775444 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.788182 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:58.830155 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:06:58.877818 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.877832 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:58.878527 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.896404 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.897354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.982373 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:58.983495 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:58.983913 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:58.985545 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:58.986255 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:06:59.032225 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:59.080020 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.085779 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.089725 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.092391 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.099694 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:59.162034 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:06:59.174396 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.175287 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.176108 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.176193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.185632 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.276695 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.279064 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.279675 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.284828 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.287628 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.370605 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.374292 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.375561 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.376850 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.385354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:59.464095 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:06:59.473464 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.473878 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.473912 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.476325 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.477937 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.562824 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.564530 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.567976 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.569684 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.573110 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:59.578900 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:06:59.596109 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:06:59.671257 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.672174 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.675123 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.682596 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.686450 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.760538 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.761894 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.763195 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.764879 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:06:59.782600 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
I20240502 14:06:59.792670 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.865360 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.866652 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.866928 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.867004 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:06:59.871318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:06:59.932354 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 14:06:59.963548 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:06:59.963917 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:06:59.964815 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:06:59.965246 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:06:59.965303 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:06:59.991340 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:07:00.011102 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:00.048542 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:07:00.062309 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:00.088129 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.088688 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.094130 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.101186 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.116740 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:00.126981 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:07:00.187103 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.187678 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.190688 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.191752 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.202551 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:00.251626 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:00.279314 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:00.279632 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:00.279798 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.281090 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.283906 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.284597 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:00.300972 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:07:00.373468 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.377928 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.379575 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.380673 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.386991 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:00.463721 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:07:00.489322 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.491569 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.492117 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.501116 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.501579 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:00.537432 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:07:00.582177 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:00.583106 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.584065 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.585875 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.588467 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.606487 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:00.609697 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 14:07:00.674360 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.678068 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.678953 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.679508 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.690572 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:00.691329 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:00.770628 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.771975 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.772392 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.773133 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.774575 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.895709 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.899085 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.905043 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.912477 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:00.914417 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:00.926724 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:07:00.964184 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:00.994855 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:00.996538 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:00.997531 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:00.997879 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:00.998224 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:01.023715 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:01.081123 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.081921 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.085347 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.087330 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.106158 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:01.112550 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:07:01.179050 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.180132 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.181433 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.184993 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.188994 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.295094 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.297143 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.297753 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.298724 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.300184 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:01.305768 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 14:07:01.347311 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:07:01.392396 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.393003 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.393429 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.394097 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.394479 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.481343 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.481858 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.483728 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.486600 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.488801 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:01.530215 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:01.531097 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:01.576747 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.578382 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.583432 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.588430 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.603253 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.691535 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.695101 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.696514 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.700739 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.705734 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:01.743374 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:07:01.783452 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.785848 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.793782 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:01.796914 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.798907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:01.869029 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:01.896106 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.900393 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.905773 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.906443 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.906652 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:01.907799 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:01.996026 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:01.997465 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:01.997572 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:01.998792 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:01.999747 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.082058 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.082595 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.085724 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:02.093161 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:07:02.094499 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.109459 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:02.115685 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240502 14:07:02.179177 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.180254 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.182405 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.188007 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.190742 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.289012 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.290906 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.292294 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.297660 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.298521 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:02.309474 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:07:02.329761 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:02.381251 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.385126 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.386018 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.394033 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.395521 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.474524 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.475939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.476833 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.477505 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.478978 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.568110 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.568526 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.572860 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.580595 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.582598 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:02.658105 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:02.681574 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.684957 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.685768 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.686120 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.687119 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.773566 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.773573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.777653 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.784147 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:02.784382 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:02.786154 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:02.807879 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:07:02.812016 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:07:02.812736 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:02.874040 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:02.874636 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.874780 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.878377 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.885017 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:02.949784 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:02.972595 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:02.973218 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:02.973278 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:02.975040 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:02.976819 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:03.018678 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:03.076648 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.077184 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.077901 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.080543 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.092820 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:03.170156 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:07:03.170847 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:03.178596 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.180305 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:03.181490 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 14:07:03.181869 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.182289 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.182898 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:03.230072 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:03.272181 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.275059 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.275872 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.277014 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.277511 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:03.307698 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:03.362526 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.366030 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.366654 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.369455 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.373237 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:03.434298 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:03.477993 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.479626 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.481621 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.481858 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.483034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:03.559768 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:03.568607 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.568692 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.570000 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.571452 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.574126 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:03.584352 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240502 14:07:03.644222 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:07:03.644884 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a75497cf22014c0cb9d079b448284b3d: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:03.668234 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.668375 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.672892 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.673233 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.676523 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.760668 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.761412 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.764878 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.765102 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.765806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.873575 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.874857 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.877303 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:03.884871 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.885689 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:03.937359 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:03.972342 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:03.973006 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:03.973971 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:03.975708 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:03.976429 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.068059 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.070245 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.070786 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.071738 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.074206 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:04.089140 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:04.162912 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.163455 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.165302 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.178742 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.180536 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:04.202088 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:07:04.269253 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.273633 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.280083 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.280913 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.281847 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.376195 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.377436 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.377897 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.378720 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.388983 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.461771 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:04.463708 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:04.463728 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:04.464329 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.464747 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.466915 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.475636 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.553674 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.554088 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.555761 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.565264 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.565477 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:04.651413 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240502 14:07:04.676312 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.677549 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.685721 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.688573 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.689024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.766369 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.769384 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:04.772907 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:07:04.776620 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.776645 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.784688 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:04.801499 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:04.866381 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.867157 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.868243 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:04.868845 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.876765 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:04.933187 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c88f85f143d845a1b7066244c606f8ba: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:04.953052 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:07:04.965469 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:04.966390 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:04.966499 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:04.967849 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:04.977057 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:05.000821 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:05.073122 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:05.076741 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.078248 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.079186 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.082597 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.099541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:05.153311 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:07:05.160759 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:05.176568 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.177359 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.177403 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:05.189306 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:05.193679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.196198 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.265846 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.266448 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.266760 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.270849 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.273878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.356624 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.360329 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.361555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:05.366037 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:05.373409 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:05.379566 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:05.379973 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.480407 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.484835 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.488663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.489774 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.496868 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:05.523732 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:05.547870 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:05.575320 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.575748 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.576071 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.580471 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.588518 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:05.602178 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 14:07:05.603063 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 14:07:05.676755 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:05.677110 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.678704 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.681228 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.681234 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.681317 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.765332 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.765434 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.765971 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.768966 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.777297 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:05.833746 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 14:07:05.874996 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:05.876210 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.878995 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.879789 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.892580 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:05.913542 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240502 14:07:05.942452 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:05.982024 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:05.983278 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:05.983494 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:05.984288 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:05.986469 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:06.072017 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:06.074860 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.074918 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.076069 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.076965 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.077885 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.169407 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.170323 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.173009 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.174052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.174985 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.276486 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.279398 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.280084 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.282135 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.289165 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.372805 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.372875 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.372875 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.395300 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.395301 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:06.403726 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:06.467998 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.472698 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.473556 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.473985 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.478193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:06.494596 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:06.522528 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:06.560897 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.562171 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.564244 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.565008 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.566272 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.674374 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.675057 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.677413 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.685319 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.686002 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:06.695166 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:07:06.764778 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.766355 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:06.766541 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.768570 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.776998 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:06.782779 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:07:06.863359 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.863904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.864564 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.864768 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.866935 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:06.889086 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:07:06.891456 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:06.957734 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:06.959121 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:06.959767 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:06.960964 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:06.974329 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:07.035364 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.069687 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.071583 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.072533 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.076325 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.082151 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:07.101878 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240502 14:07:07.165328 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.165787 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.168419 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.168944 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.169191 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:07.182731 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.255553 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.258459 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.259322 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.259516 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.272639 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.348141 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.350333 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.350652 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.353171 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:07.366706 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.367945 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:07.394685 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:07:07.453444 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:07.458168 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.461000 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.466106 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.469271 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.477771 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.552634 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.553205 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.553270 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.557013 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.557998 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:07.628517 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:07.654580 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.654819 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.656227 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.668757 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.669365 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.747687 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.748742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.749080 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.750774 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.751215 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:07.846891 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.856763 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:07.859779 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.865804 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:07.866374 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.867075 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.867136 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:07.909727 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:07.961812 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:07.965507 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:07.965767 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:07.967693 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:07.969832 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:08.018422 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 14:07:08.058491 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:08.059335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.059345 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.060230 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.062871 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.063889 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:08.136127 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:08.154290 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.154345 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.155973 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.158507 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.164793 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:08.172279 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:07:08.205464 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:07:08.206285 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:08.262971 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.266157 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.266551 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.268873 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.269511 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:08.356539 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:08.361753 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:08.363636 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 14:07:08.365139 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.368816 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.379180 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.388145 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:08.453370 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:08.457234 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.460884 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.462814 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.465165 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.471839 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.549613 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.551946 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.552803 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.553400 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.564085 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:08.565222 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:07:08.676244 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.676652 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.679939 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.680584 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.686326 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.768669 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.772127 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.773900 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.781096 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.794212 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.869666 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.871143 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.872344 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.874425 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:08.875710 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.973239 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:08.976066 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:08.977088 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:08.977334 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:08.977607 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:08.997983 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:09.082950 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.084029 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.085204 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.086635 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.095360 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.179522 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.180027 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.184381 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.184959 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.198422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:09.204149 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:09.276578 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.280370 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.282285 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.289917 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.294613 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:09.361522 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:07:09.375648 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.381402 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.381934 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.389643 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.398790 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:09.434273 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:09.434599 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
I20240502 14:07:09.483609 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.484311 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.486335 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.489188 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.496560 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:09.497538 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:09.575568 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.575955 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.580050 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.584481 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.600384 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.680261 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.681972 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.682931 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.692381 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.695655 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:09.736168 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:09.775606 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.776916 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.777516 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.779812 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.780448 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:09.833626 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 14:07:09.880841 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:09.884814 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240502 14:07:09.891026 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.892179 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.892251 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.892463 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:09.914865 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.997440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:09.997754 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:09.998333 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:09.999325 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:09.999714 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:10.064307 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:07:10.089807 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.095134 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.096765 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.108541 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:10.113550 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.155349 30736 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:10.155467 30736 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:10.156061 30736 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:10.156463 29028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:07:10.156632 29028 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:07:10.156843 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:07:10.157613 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:10.157750 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:10.157801 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:10.157963 30736 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:10.201931 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.204233 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.204850 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.205206 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:10.207654 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:10.221433 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.301021 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.302589 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.304745 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:10.305747 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:10.306169 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.307044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:10.330897 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:10.345494 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.391736 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.394143 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.394198 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.395669 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.410102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:10.466797 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.502003 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.502975 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.503670 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:10.504993 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.505589 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:10.581432 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.592818 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.595965 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.597862 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:10.598222 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.609954 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.691130 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.692572 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.695032 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.695505 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.696447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:10.705307 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:10.776348 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.779672 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:10.787959 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:10.788311 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.790726 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:10.794926 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:10.810094 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240502 14:07:10.895545 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.901289 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:10.901420 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:10.906374 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:10.912815 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:10.913208 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:10.955266 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:10.995765 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:10.998337 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.001559 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.002905 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.003722 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.003747 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.091104 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.091722 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.092128 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.093679 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:11.094028 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 14:07:11.098018 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.185092 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.185721 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.186502 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.190155 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.210901 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.293802 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.295503 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.297462 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.298887 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.299180 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:11.379444 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:11.392184 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.393443 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.394577 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.397323 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.400427 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.487295 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.488528 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.488605 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.490315 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.493196 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.573688 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.575752 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.575783 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.580435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.593673 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:11.612880 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:11.652258 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:07:11.688848 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.689738 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.689960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.692067 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.698710 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:11.734915 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
I20240502 14:07:11.774753 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.783380 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.785014 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.793177 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.813180 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:11.864231 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:11.881390 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:11.901705 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.902699 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.905050 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:11.908099 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.914563 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:11.996582 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:11.996698 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:11.997143 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:11.998991 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.009661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:12.022790 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:12.026417 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:07:12.113201 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:12.119549 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.120185 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.120716 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.124184 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.136786 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:12.184962 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:12.201576 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:12.212397 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.215277 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.215530 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.217695 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:12.218130 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.298159 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.299101 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.299456 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.301401 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.314896 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:12.368093 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:12.389186 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.394492 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.394709 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:12.398779 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.400447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.517560 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.519636 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.519919 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.521543 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:12.526397 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.604070 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.604774 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.611569 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.615072 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.626880 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:12.656957 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:12.695302 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.697319 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.697438 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:12.697691 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.699680 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.788725 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.790510 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.791801 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.792520 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:12.793666 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:12.821870 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:12.835865 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:12.905745 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:12.918100 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:12.919126 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:12.920189 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:12.921500 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:12.934223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:12.938113 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:12.985848 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:13.010277 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.011178 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.011803 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.016619 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.019891 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.099730 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.101497 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.108851 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.111651 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.113451 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:13.181489 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:13.195708 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.197158 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.198324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.200939 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.201542 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:13.256723 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:13.262354 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240502 14:07:13.287146 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:13.293746 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:07:13.294535 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 14:07:13.310349 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.312599 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.313542 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.313817 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.314215 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:13.323901 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:13.400178 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.404628 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.405772 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.406193 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.414165 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:13.454195 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:13.492187 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.495488 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.496567 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.498256 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.504666 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.572809 358 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:13.572948 358 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:13.573468 358 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:13.574554 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:07:13.574697 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:07:13.575245 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:13.575335 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:13.575371 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:13.575577 358 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:13.581667 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.584189 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.585444 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.588905 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.593163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.693163 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.694013 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.695991 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.697707 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.699205 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:13.745003 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:13.789649 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.793113 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.793447 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.795333 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.796991 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.890305 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.890877 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.891137 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.892343 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:13.895741 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.985558 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:13.990141 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:13.991612 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:13.993227 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:13.993362 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:14.062356 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240502 14:07:14.090981 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.090977 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.091516 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.096753 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.100757 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:14.122224 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:14.185129 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.189378 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.189900 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.190361 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.197790 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:14.241134 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 14:07:14.266243 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
I20240502 14:07:14.286141 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.288172 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.289254 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.300128 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.302112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:14.349936 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:14.377506 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.379148 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.381767 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.382601 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.397872 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:14.457716 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:14.477684 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.479193 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.480384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.482106 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.488420 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.565459 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.567569 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.567667 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.576081 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.576164 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:14.604947 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:07:14.654556 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240502 14:07:14.670258 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:14.676952 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.677814 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.677842 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.687116 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.688436 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.767199 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.767882 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.770613 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.780556 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:14.787127 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.873301 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.874362 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.875627 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.876746 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.879300 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:14.937590 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:14.975692 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:14.978623 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:14.978696 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:14.979143 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:14.980309 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:15.068892 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:15.076057 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.076540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.078163 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.078708 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.080864 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:15.096983 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:07:15.122398 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 14:07:15.168303 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.170138 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.171733 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.177368 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.177819 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:15.269075 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240502 14:07:15.270078 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:15.272854 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.277771 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.278050 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.289170 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.289355 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.364395 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.365453 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.374444 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.374526 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.380605 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:15.446684 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:15.469527 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.470752 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.472229 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.472896 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.473163 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.558121 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.562232 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.563949 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.566745 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.572002 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:15.578624 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:07:15.593175 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:15.670310 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.671051 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.675570 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.681229 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.691881 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:15.705381 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:07:15.715493 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:15.765606 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.766120 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.766918 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.769335 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:15.775844 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 14:07:15.789572 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:15.860481 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 14:07:15.868602 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:15.869748 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.871711 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.873118 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.885214 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:15.941869 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:15.965068 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:15.966053 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:15.968577 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:15.972970 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:15.987918 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.066489 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.066943 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.071880 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.076486 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.077456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.158406 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.159854 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.165129 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.165658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.168363 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:16.185842 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 14:07:16.197743 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:16.266444 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.266484 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.269907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.270351 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.282248 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.358214 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.362285 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.362777 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.363533 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.365123 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.452996 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.454473 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.457218 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.465498 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.466068 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.547839 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.548731 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.548803 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.559348 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.564425 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:16.576094 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:07:16.614586 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:16.624866 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240502 14:07:16.637858 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:16.656117 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.660966 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.661123 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.661876 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.662145 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.747021 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.750326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.750586 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.753562 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.761646 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:16.791539 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:07:16.792172 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:07:16.801509 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240502 14:07:16.854353 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.861769 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.861832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.867923 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.868362 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:16.957507 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:16.960954 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:16.961270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:16.961840 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:16.962239 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:17.011900 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:17.048763 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.049928 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.050524 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:17.053473 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.053943 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.144088 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:17.145025 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:07:17.149607 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.149792 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.157122 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.157523 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:17.229205 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:17.261013 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.261127 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.263501 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.264748 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.269234 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:17.349654 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:07:17.351658 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.351874 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.354094 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.354784 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.355454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:17.416002 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:17.443790 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.445868 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:17.446005 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.453258 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:17.453296 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:17.463694 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.539574 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.540050 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.540165 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.540215 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.541399 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:17.569787 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:17.663789 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.665158 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.668852 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.668884 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.684005 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:17.735703 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:17.756028 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.757141 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.758095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:17.759810 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.760149 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:17.853215 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:17.861295 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.863541 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.863628 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.863974 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:17.864462 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:17.959795 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:17.961702 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:17.964176 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:17.964880 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:17.972441 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:18.006170 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:07:18.010573 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:18.052727 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.054556 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.057194 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.059384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.065073 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:18.076197 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:18.145278 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.147169 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.148051 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.150650 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.165804 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:18.173038 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 14:07:18.205072 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:07:18.261194 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.263326 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.264238 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.269980 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.274644 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.352463 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.355778 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.356155 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.360569 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.368645 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.448465 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.452440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.453758 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.454306 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.455384 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:18.464160 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240502 14:07:18.464898 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 14:07:18.535326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.537892 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.541852 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.542577 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.549090 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:18.564141 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:18.609761 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:18.650753 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.652139 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.655858 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.656423 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.660418 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:18.725229 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:18.737792 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.739054 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.749478 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.750816 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.752928 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.839162 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.839188 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:18.844305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.845682 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.855118 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.931299 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:18.936480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:18.937995 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:18.938910 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:18.957741 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.032071 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:19.034425 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:19.034440 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.035161 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.035768 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.046495 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:19.046691 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:19.119390 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.122013 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.122638 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:19.123230 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:19.125020 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.132691 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:19.156790 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:19.230988 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:19.232932 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:07:19.233002 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.234418 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.235456 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.238281 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:19.272230 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240502 14:07:19.320071 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.321609 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.323134 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.326400 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.334064 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.417060 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.417383 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.417778 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.417961 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.421005 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:19.475016 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:07:19.504026 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.505782 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.507207 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.507470 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.512424 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.623450 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.627530 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.627807 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.632587 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.633596 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:19.657580 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:07:19.658492 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:19.726436 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.728853 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.729377 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.730345 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.730382 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:19.745759 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:19.823792 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.828125 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.828581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:19.830063 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.837332 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.931157 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:19.931281 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:19.940559 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:19.941818 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:19.945044 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.026822 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.027154 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.033664 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.034756 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.035897 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:20.046860 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:20.118407 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:20.125749 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.126200 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.126950 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.127012 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.127959 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:20.146574 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 14:07:20.147533 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:20.246727 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.247009 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.247015 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.249823 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.252928 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:20.278342 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 14:07:20.312871 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:20.332336 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.336575 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.336689 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.339366 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.353751 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.429919 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.433746 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.437100 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.437558 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.447865 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:20.486708 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:20.521765 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.524818 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.526592 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.527299 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.528261 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:20.607144 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:20.640255 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.641165 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:20.641927 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:20.642582 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.644034 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.648471 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:20.701176 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:07:20.733624 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.734331 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.736397 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.739005 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.747604 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:20.824352 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:20.830653 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:20.832072 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.832440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.833534 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.840253 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:20.861665 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 14:07:20.935143 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:20.940748 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:20.941586 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:20.941973 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:20.953428 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.027643 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.030586 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.032613 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.034220 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.042294 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:21.073158 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:07:21.094981 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:21.126495 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.127172 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.127523 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.135778 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.136031 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.244900 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.246913 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.247004 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.248104 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.257354 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:21.324703 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:21.336887 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.340526 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.341256 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.341518 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:21.350543 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:21.350790 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:21.430687 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:21.439453 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.440387 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.440757 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.442907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.456426 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.528584 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.528882 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.531927 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.539351 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.550930 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:21.599323 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:07:21.624351 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:21.644240 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.648753 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.649003 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.651348 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.651592 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.740981 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.741894 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.743665 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.756104 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.756263 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:21.778496 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240502 14:07:21.815735 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 14:07:21.832975 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.833127 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:21.842080 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.844005 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.858709 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:21.902113 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:07:21.936487 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:21.937846 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:21.938228 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:21.939322 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:21.939518 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:22.014680 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240502 14:07:22.043422 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.045758 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.047129 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.048245 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.067036 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:22.137432 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:22.138190 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.140537 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.146463 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.150183 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.150393 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:22.201845 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:22.238323 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.239707 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.241523 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.248895 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.250674 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:22.315342 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:22.327627 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.328125 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.331466 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.336584 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.352906 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.437394 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.440135 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.445062 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.445511 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.445863 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.534564 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.535619 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.537484 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.540525 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.549973 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:22.620714 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:22.622182 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:22.634691 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.637673 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.639061 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.640573 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.641155 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:22.720633 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:22.728240 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.728549 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.729175 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.730497 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.730656 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:22.797212 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:07:22.809273 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:07:22.812878 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 14:07:22.839839 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.845643 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.848740 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.856540 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.858620 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:22.957548 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:22.960345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:22.961258 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:22.962517 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:22.972543 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.055790 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.056218 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.056979 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.062220 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.069398 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:23.148622 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:23.152245 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.153504 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.155663 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.155887 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:23.156301 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:23.156584 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.260490 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.260720 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.261065 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.262364 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.267457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:23.277489 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240502 14:07:23.321931 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:23.348008 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.349372 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.351897 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.352344 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.371915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.448365 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.451579 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.452031 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.454638 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.460726 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:23.539036 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:07:23.541661 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.542575 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.546643 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.548255 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:23.556227 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:23.560698 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:23.573719 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 14:07:23.632339 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:23.652386 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.653740 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.654546 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.655766 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.663633 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.739456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.742691 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.744226 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.744264 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.745568 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:23.796336 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:07:23.815975 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:23.847505 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.850546 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.852456 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.854390 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.870954 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:23.956753 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:23.956763 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:23.960400 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:23.960777 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:23.964172 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:23.999405 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:07:24.044760 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:24.049093 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.049924 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.051859 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.052743 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.054334 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.137022 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.137477 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.138003 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.148577 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.151049 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:24.160777 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:24.190712 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:07:24.257228 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.258860 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.260043 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.261516 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.264425 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.348491 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.350378 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.351836 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.352456 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.352936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.439826 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.441686 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.442574 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.443475 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.445077 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:24.495060 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240502 14:07:24.536450 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.536597 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.537530 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.540910 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.546918 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:24.568955 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:07:24.569443 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:24.589565 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 14:07:24.616811 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:07:24.651870 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.652806 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.655299 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.661780 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.662339 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.739145 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.741801 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.742161 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.742523 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.751365 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.839792 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:24.840058 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.845569 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.857370 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.858024 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.933476 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:24.936950 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:24.939412 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:24.946406 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:24.953292 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.036985 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.042649 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.043390 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.052448 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.054718 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:25.067319 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:25.132751 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.133314 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.134542 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.145915 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.146243 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:25.196899 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 14:07:25.232292 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:25.234759 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.235788 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.239584 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.245169 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.245448 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:25.266395 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:25.322962 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.324425 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.326673 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.327299 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:25.332537 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:07:25.341226 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:25.346302 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:25.433387 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.434495 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.437816 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.438117 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.445240 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:25.493407 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:07:25.506458 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240502 14:07:25.524811 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.526736 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.529513 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.530830 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.532606 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:25.564157 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:25.621521 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.626066 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.630232 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.633818 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.638962 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:25.668654 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:25.719923 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.720477 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.722743 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.722815 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.725666 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.822072 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:25.824116 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.827172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.830225 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.835278 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.924186 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:25.925009 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:25.926285 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:25.927345 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:25.929942 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:25.963019 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:26.018834 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.020011 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.022033 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.022214 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:26.029697 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:26.031672 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:26.037400 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:26.075073 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 14:07:26.117612 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.118290 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.120956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.121552 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.128598 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:26.131711 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:26.223493 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.225883 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.226439 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.227115 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.230082 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.311637 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.313716 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.314179 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.316546 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.321681 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:26.381345 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:26.420529 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.421770 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.421770 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.422168 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.423628 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:26.508322 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:26.508656 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.509294 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.517400 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.520426 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.523056 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:26.531430 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:07:26.558127 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:26.607188 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.610642 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.611611 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.621078 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.621266 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.699198 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.700973 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.703850 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.705863 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.708317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.807873 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.809540 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.812522 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.812522 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:26.812922 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:26.825325 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240502 14:07:26.859126 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:26.894856 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:26.913573 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:26.915606 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:26.915928 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:26.916312 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:26.916337 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.008774 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.009596 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.011567 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.013783 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.032202 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:27.039899 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:27.110941 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.114326 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.116148 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.117862 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:27.126130 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:27.132889 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:27.186234 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:27.214187 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.215030 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.218420 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.219672 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.228510 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.310170 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.310997 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.311300 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.311897 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.321652 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:27.410928 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:07:27.415827 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.416177 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.416643 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.419755 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.420554 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.505771 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.507762 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.508231 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.517515 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.525396 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.606228 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.606950 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.607546 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.611095 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:27.612784 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:07:27.621271 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:27.656598 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:07:27.657196 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:27.657871 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:27.711982 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.713361 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.714255 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.715785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:27.725590 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.808413 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.808588 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:27.809326 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.810756 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.812072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:27.871013 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 14:07:27.902621 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:27.912820 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:27.913920 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:27.914528 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:27.922206 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:27.926344 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.015647 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.015877 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.017931 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.024664 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.031396 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:28.099344 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:07:28.099959 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:28.108843 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.110401 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.114759 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.118953 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.124428 5036 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:28.124536 5036 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:28.125043 5036 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:28.125445 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:07:28.125584 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:07:28.127858 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:28.127949 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:28.127983 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:28.128316 5036 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:28.134850 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:28.213713 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:28.216701 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.218873 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.218887 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.220629 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.224422 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.307960 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.308646 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.309830 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.311213 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.311605 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.415645 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.417789 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.418681 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.419878 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.424930 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:28.476470 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:28.505568 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.505992 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.508231 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.513604 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.522516 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:28.544129 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:28.605708 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.605892 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.609777 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.611388 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.621235 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:28.622563 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:28.702750 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.702975 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.704380 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:28.704804 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.704892 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:28.711069 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:07:28.711771 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 14:07:28.789052 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:28.812531 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:28.814350 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:28.815479 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.815694 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.816985 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.817760 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:28.918982 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:28.926635 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:28.927868 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:28.929371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:28.929668 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:28.936853 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:29.000885 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
I20240502 14:07:29.024325 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.025244 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.027218 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.027997 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.032444 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.120151 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.122289 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.122320 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.124608 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.130108 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.229573 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.233067 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.234633 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.234834 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.236346 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:29.314249 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:29.323848 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.324752 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.324752 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.325024 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.333657 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.419996 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.421332 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.423384 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.423813 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.423858 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.505254 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.508533 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.510772 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.511046 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.520941 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:29.526801 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:29.627822 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.628242 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.634174 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.637923 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.638010 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:29.653364 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240502 14:07:29.653556 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 14:07:29.713723 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:29.718674 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.719219 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.721724 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.722812 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.737519 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:29.802021 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:07:29.824632 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.829438 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.829790 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:29.830369 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.846554 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:29.896000 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:07:29.926765 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:29.927810 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:29.928690 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:29.930981 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:29.935325 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:29.948693 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:30.028250 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.029609 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.030282 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.033393 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.040102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.127319 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.127411 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.127601 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.130393 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:30.145190 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:30.147301 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:30.229316 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:30.236699 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.239101 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.240871 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.241338 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.243604 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:30.260911 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 14:07:30.334805 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.335776 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.339414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.339414 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.345780 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:30.377475 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:07:30.427124 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:30.427597 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.427917 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.428194 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.430558 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:30.437523 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 14:07:30.443038 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:30.483908 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:30.515112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.515354 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.518173 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.519706 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.522122 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:30.583140 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 14:07:30.611109 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:30.630546 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.631732 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.634007 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.634675 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.653357 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.730836 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.732597 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.732802 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.742615 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:30.744918 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:30.748970 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.835467 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.836001 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.836462 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.845695 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.847252 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:30.934998 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:30.935969 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:30.942618 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:30.944577 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:30.960613 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:30.998638 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:07:31.001111 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:07:31.004900 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:31.043035 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.046633 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.046648 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.047449 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.049639 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:31.083360 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:31.119057 5036 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:31.119206 5036 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:31.119674 5036 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:31.119951 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:07:31.120101 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:07:31.120750 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:31.120857 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:31.120918 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:31.121064 5036 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:31.143599 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.144565 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.144994 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.145107 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.156198 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:31.197202 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 14:07:31.245339 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.247437 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.249073 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.249415 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.250907 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.336627 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.337967 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.340998 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.341709 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.362520 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:31.395231 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
I20240502 14:07:31.438102 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.440461 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.440912 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.441666 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.444016 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:31.447913 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:31.528102 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.531837 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.532298 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.534662 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.536415 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:31.588325 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:31.648295 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.649942 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.650133 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.656806 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.660557 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.739075 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.740300 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.740804 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.744613 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.752573 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.837226 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:31.837487 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:31.841230 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.848513 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.848963 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:31.911038 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:07:31.937856 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:31.938478 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:31.938849 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:31.941974 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:31.946538 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:31.958724 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:32.006824 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:32.038415 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.040640 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.041813 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.042336 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.048398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.127683 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.127784 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.131207 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.133638 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.148535 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:32.191869 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240502 14:07:32.224026 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:07:32.239209 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.239809 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.241947 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.243793 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.245955 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:32.275127 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:32.328636 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.328868 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.329896 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.330626 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.333366 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:32.396185 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:32.425244 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.427170 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.429836 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.431295 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.432646 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:32.442258 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:07:32.511698 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.514158 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.514607 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.516409 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.524840 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:32.570674 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:32.627245 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.628969 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.633508 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.634896 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.643357 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:32.668102 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:32.719161 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.719755 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.721241 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.723176 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.725370 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.813891 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.814664 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.819394 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.819516 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:32.821424 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.921357 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:32.921752 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:32.923103 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:32.924860 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:32.944424 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:32.999667 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:33.015043 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.017133 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.017203 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.017679 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.020305 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:33.057155 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:33.057824 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:07:33.073190 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:33.108870 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.112076 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.115442 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.123118 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:33.133213 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:33.134586 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:33.184760 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:33.227695 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.229821 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:33.232242 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:33.232764 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.241016 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.241570 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.324469 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.325680 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.326705 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.331260 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.342397 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.423192 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.423949 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.425103 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.425199 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.432623 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:33.503242 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:07:33.507002 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:33.515671 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.516809 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.518126 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.519738 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.520496 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:33.620350 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:33.632397 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.633415 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.633563 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.633818 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.636654 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:33.684628 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:33.691172 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:33.702387 5036 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:33.702503 5036 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:33.702913 5036 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:33.704303 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:07:33.704443 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:07:33.705271 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:33.705379 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:33.705421 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:33.705667 5036 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:33.720332 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.721172 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.722092 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.724313 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.732496 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:33.733078 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240502 14:07:33.798552 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 14:07:33.811933 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.812196 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:33.813773 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.824654 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.825158 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.920866 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:33.920977 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:33.921643 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:33.926066 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:33.933179 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:34.004814 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:07:34.016646 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.020226 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.020962 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.022809 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.034857 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.110921 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.117333 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.118168 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.124960 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.128054 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:34.172755 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:34.215759 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.216414 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.218204 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.219488 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.224808 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.309296 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.311303 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.311709 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.312705 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.316783 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:34.357260 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:34.409171 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.414623 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.414695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.422223 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.423784 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.500268 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.503340 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.504364 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.505131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.512804 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.603744 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.604379 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.605159 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.616415 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.621140 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:34.664242 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:34.699905 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.700083 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.703033 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.705551 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:34.713189 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.801025 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.802726 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.803791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:34.804188 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240502 14:07:34.804602 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.813323 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:34.873173 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:07:34.897095 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:34.909469 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:34.911196 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:34.917834 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:34.918036 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:34.925642 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:07:34.963013 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:35.002058 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.002238 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.005800 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.006332 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.017365 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.090557 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.091940 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.100426 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:35.100801 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:07:35.111176 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.119691 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:35.215514 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:35.219702 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.220229 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.224521 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.224639 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.224954 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:35.254452 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:35.308892 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.310158 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.311250 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.312369 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.314507 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:35.317063 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:07:35.342119 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 74 similar messages]
I20240502 14:07:35.402288 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.402452 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.403916 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.404345 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.405817 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:35.476024 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:35.494410 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.498149 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.498965 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.508100 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.508414 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:35.590179 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:07:35.603893 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.608330 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.609473 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.616145 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.627015 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:35.635556 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 14:07:35.694851 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:35.705046 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.705808 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.708089 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.715811 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.716480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:35.754441 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:07:35.763742 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:35.801384 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.803416 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.805476 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.807648 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:35.831777 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:35.839998 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:35.905843 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:35.907886 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:35.912912 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:35.918669 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:35.925470 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:35.925714 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.012028 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.012842 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:36.014470 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.015825 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.016314 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:36.042992 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:07:36.092358 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:36.101439 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.103015 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.108587 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.109547 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.116981 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:36.123334 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 14:07:36.216755 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.217265 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:36.217527 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.219357 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.220127 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.310339 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.311226 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.312829 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.313966 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:36.329439 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:36.364833 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:36.404659 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.406075 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.407289 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.409442 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.410930 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:36.457633 28869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:07:36.458400 28869 catalog_manager.cc:4573] TS 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a75497cf22014c0cb9d079b448284b3d: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:36.493233 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.493964 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.496210 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.499303 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:36.509747 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.610563 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.610800 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.620957 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.625077 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.628154 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:36.656211 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240502 14:07:36.701890 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.705034 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.705461 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.713675 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:36.716526 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.797591 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.800668 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.804034 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.804744 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.811131 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:36.814164 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:36.894632 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:36.902750 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:36.902990 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:36.904630 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:36.906972 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:36.908601 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.000950 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.006065 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.006217 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.007300 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.014408 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.097602 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.098289 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.099062 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.099856 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.100705 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.200268 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.201967 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.207544 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.209666 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:37.219312 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:37.222220 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:37.222841 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:07:37.293323 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.295049 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.295495 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.298907 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.300112 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:37.308856 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:07:37.323156 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:07:37.343428 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240502 14:07:37.391754 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:37.393728 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.397500 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.397922 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.399176 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.408491 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.481909 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.484259 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.486449 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.486914 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.487751 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.591655 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.591964 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.596730 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.604305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.605075 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:37.605464 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:07:37.622646 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:07:37.691658 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.692667 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.693128 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.696027 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:37.708217 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c88f85f143d845a1b7066244c606f8ba: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:07:37.715252 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:37.744601 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:37.780362 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.783311 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.786659 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.786756 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.787372 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:37.827879 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:37.882180 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:37.882898 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.882998 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.887087 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:37.896557 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:37.998613 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:37.998682 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:37.998925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.001296 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.002599 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.090704 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.091634 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.094038 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.094791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.095178 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:38.141325 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:38.141935 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:38.175210 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 14:07:38.190467 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.192658 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.193292 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.193861 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.205768 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:38.231544 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:38.283113 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:38.283833 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.287288 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.296739 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.296767 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.299119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:38.305270 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 14:07:38.386148 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:38.394227 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.398557 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.400466 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.405015 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.417192 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.491708 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.492880 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.492935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.493578 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.493947 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.585880 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.589627 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.593312 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.596072 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.601711 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:38.663064 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:38.678757 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.679272 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:38.680701 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:38.682623 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.682838 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:38.686913 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:38.696045 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:38.727316 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 14:07:38.753738 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:38.793818 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.794499 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.801141 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.808413 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.816094 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.893296 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.896659 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.900601 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:38.900686 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.908452 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:38.940495 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240502 14:07:38.972752 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:07:38.987447 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:38.988102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:38.988178 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:38.991086 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:38.992058 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.075933 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.077183 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.078543 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.083628 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.092172 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:39.140018 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:39.192205 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.196637 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.200448 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.201186 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.201331 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:39.269177 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:39.283336 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.284514 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.287020 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.294083 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.300766 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.379163 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.379195 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.386868 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.399139 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.401270 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.475220 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.480111 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.480773 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.481442 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.482355 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:39.501488 28869 catalog_manager.cc:4573] TS 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 015b587864d54da8aee36c3afd968e86: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
I20240502 14:07:39.582520 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.583971 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.584136 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.584735 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.586283 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.674656 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.679790 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.680570 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.682189 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.696497 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:39.714604 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:07:39.768246 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.770093 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.776185 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.776486 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.781726 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:39.840991 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:39.854802 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:39.868137 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:39.872813 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.872795 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.872997 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.874848 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:39.875638 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:39.972998 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240502 14:07:39.975548 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:39.978425 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:39.980001 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:39.980249 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:39.980836 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:39.982599 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:07:40.066198 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.066212 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.068235 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.069116 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.070996 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:40.091145 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:40.129848 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:40.174011 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.174577 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.174819 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.178391 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.194932 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.262993 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.263273 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.264840 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.266171 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.268455 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:40.339875 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:40.371719 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.377777 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.377939 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.380582 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.386818 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:40.451014 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:40.463636 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.463991 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:40.466275 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 14:07:40.468218 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.468917 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.469544 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.573166 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.575016 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.575759 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.577898 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.584725 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:40.614575 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 14:07:40.615250 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:40.663640 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.664665 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.667376 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.668409 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.668846 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.770659 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.773903 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.775285 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.778826 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.781150 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:40.861450 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:40.872350 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.874147 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:40.874292 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:40.874516 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:40.876122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.878767 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.981469 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:40.983338 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:40.985776 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:40.997786 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:40.998082 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:41.010192 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:41.028923 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:41.064394 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:41.079810 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.080338 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.084461 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.085989 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.105791 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.181102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.181186 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.183691 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.185618 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.193135 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.273777 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.274915 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.275858 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.276477 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.276901 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:41.285568 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:07:41.334259 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 14:07:41.386680 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.390237 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.390640 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.391726 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.393393 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:41.406939 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:07:41.433502 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:07:41.483243 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.483243 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.483757 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.486629 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.500670 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:41.521387 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:41.526432 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:41.574918 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.577159 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.578414 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.578917 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.583079 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.674052 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.675720 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.676488 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.677093 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.681447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.783180 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.785248 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.787529 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.796873 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.810882 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:41.889487 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.889919 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:41.891685 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.894815 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.896317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:41.935880 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:41.990981 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:41.993057 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:41.993129 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:41.993925 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.009639 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.082281 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.082502 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.084321 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.085753 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:42.103924 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:07:42.105372 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.193775 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.193933 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.197830 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.210191 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.217360 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.283923 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.287032 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.287619 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.288197 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.289964 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.388504 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.390093 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.392724 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.401841 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.402771 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:42.422336 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:42.424293 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 14:07:42.430696 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:42.430682 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:42.488505 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.488992 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.490998 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.491631 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.500587 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:42.510376 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:42.584234 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.584509 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:42.585098 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:07:42.589157 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:42.593060 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:42.598909 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.606123 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:42.668572 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
I20240502 14:07:42.679456 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.683929 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.684444 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.684700 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.693826 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.788790 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.788892 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.791042 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.796424 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:42.804764 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:42.864532 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:42.885190 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.887625 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.889672 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.889914 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.905920 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:42.965401 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:42.984338 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:42.985117 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:42.987799 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:42.993067 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:42.998107 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:42.999272 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:43.077631 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.077723 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.078969 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.081277 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.081848 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:43.189198 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.189291 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.195488 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.200744 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.200805 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:43.278137 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:43.282394 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.283613 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.285218 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.285261 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:43.290409 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.371609 10681 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:43.371755 10681 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:43.372371 10681 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd692a8958c241fca3aec5cca3a0ded5 (127.16.217.3:41345), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:43.372824 29293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "a7221b90d0aa4770a8ce5522103a96f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "cd692a8958c241fca3aec5cca3a0ded5" is_pre_election: true
I20240502 14:07:43.372963 29293 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7221b90d0aa4770a8ce5522103a96f4 in term 2.
W20240502 14:07:43.374519 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:43.374646 28961 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:43.374694 28961 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:43.374816 10681 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:43.384091 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.389329 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.389847 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.390321 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.392720 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:43.442961 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 14:07:43.465795 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:43.466499 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:43.475546 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:43.478710 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.480438 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:43.481662 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.484333 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.484701 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:43.545251 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:07:43.577792 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.581390 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.582702 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.583259 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.595067 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:43.656726 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:43.671119 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.673605 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.676656 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.684939 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:43.692996 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:43.742674 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:43.776422 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.780493 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.780988 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.788581 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:43.789606 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:43.793857 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:43.853041 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 14:07:43.876971 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.878871 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.880774 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:43.882448 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.885643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:43.885996 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:43.981021 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:43.981304 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:43.986073 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:43.986202 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:43.992889 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:44.037997 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240502 14:07:44.071930 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.072067 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.073335 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.073478 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.083577 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.178640 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.180905 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.181845 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.183507 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.183821 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:44.208813 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:44.270864 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.271266 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.274680 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.280790 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.281006 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.368955 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.372907 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.376061 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.376745 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.377252 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.461635 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.462661 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.463593 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.465330 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.472047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:44.485894 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:44.570631 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.572736 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.574453 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.574818 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.585090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.661846 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.663782 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.664698 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.667014 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.673595 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.755730 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.755978 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.756125 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.756469 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.758806 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.843698 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:44.846418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:44.848198 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.854849 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.869972 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:44.881712 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:07:44.892084 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:44.931813 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:44.960320 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:44.960676 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:44.965893 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:44.966603 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:44.968741 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:44.979555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.055580 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:45.058455 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:45.059635 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.060742 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.061230 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.062122 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:45.101409 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:07:45.147616 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.150687 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.150689 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.151533 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.153851 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:45.166328 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:45.247457 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.249686 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.252305 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.252413 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.265340 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:45.349670 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:45.360903 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.368737 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.373997 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.379045 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:45.382462 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:45.384145 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.409094 11771 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:45.409221 11771 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:45.409688 11771 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:45.409935 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:07:45.410094 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:07:45.410800 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:45.410907 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:45.410952 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:45.411079 11771 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:45.458050 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.458766 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:45.459069 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240502 14:07:45.459192 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.460898 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.464438 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.550056 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.551873 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.551973 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.553097 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.554361 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:45.578207 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 14:07:45.580090 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:45.646062 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.646449 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.646708 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.655529 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.659536 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.760725 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.764140 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.766069 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.767856 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.778185 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:45.808109 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:07:45.839284 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:07:45.874249 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:45.875314 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.875861 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.878710 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.879767 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:45.913736 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:45.974889 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:45.977157 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:45.977936 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:45.978171 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:45.979142 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:46.016208 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:07:46.024178 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:46.067938 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.068012 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.070413 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.070643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.079488 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:46.139642 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
I20240502 14:07:46.184726 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.186177 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.187214 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.187285 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.188457 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:46.218634 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:07:46.220868 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:46.272080 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.273629 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.275871 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.277364 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.285025 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:46.348551 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:46.377015 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.378876 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.382117 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.382692 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.388864 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:46.391108 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 14:07:46.467327 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.469707 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.470679 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.470947 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.471498 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:46.487919 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:46.568133 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.568598 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.572808 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.573659 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.576364 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.661017 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.662919 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.663512 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.665246 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.665634 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:46.693770 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:46.770007 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.771191 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.771899 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.773190 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.784904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:46.817925 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:46.876127 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.877321 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:46.877835 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.884411 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.894234 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.972003 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:46.973872 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:46.981179 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:46.983588 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:46.983981 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.065169 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.067355 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.068588 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.068787 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.072438 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.173269 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.176090 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.181078 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.185609 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.186913 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:47.236459 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240502 14:07:47.240818 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:47.268163 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.268450 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.269268 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.270030 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.271636 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.363598 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.369369 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.372694 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.380722 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.383050 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.454377 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.457926 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.458850 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.460561 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.468734 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:47.486763 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:07:47.560065 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:47.566695 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.568166 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.569078 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.572429 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.580682 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:47.611939 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:47.612802 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:07:47.625512 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:47.661273 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.662735 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.665431 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.667062 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.686369 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:47.702348 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240502 14:07:47.748785 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:07:47.760874 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.764456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.766436 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.767184 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.783818 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:47.821122 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:47.861284 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.867127 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:47.867126 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.869357 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.881907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:47.889726 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 14:07:47.974370 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:47.975090 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:47.979161 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:47.979523 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:47.980494 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.071427 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.071482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.072486 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.074438 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.085505 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.161670 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.164440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.165570 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.166236 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.166813 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.255511 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.256003 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.257586 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.258148 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.258435 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:48.321928 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:48.366339 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:48.367384 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:48.367722 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:48.368736 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:48.369848 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.369876 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.380564 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:48.444326 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:48.460158 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.461591 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.462055 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.462145 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.462561 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:48.470837 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:48.552057 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.554670 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.563455 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.564097 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.564096 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.641901 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.643572 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.645557 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.645956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.653645 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:48.716030 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 14:07:48.751240 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:07:48.753379 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.755242 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.755373 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.756124 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.758924 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.850950 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.853492 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.854923 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:48.855393 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.855643 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:48.879195 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:07:48.879838 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:48.888680 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 14:07:48.905835 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:48.939057 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:48.958626 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:48.963871 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:48.964504 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:48.968400 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:48.978286 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.053040 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.053133 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.056372 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.057513 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.058563 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.152004 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.152544 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.153687 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.157398 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.173486 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.245867 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.249267 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.250391 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.252631 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.256403 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.345930 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.348374 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.351569 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.351601 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.363806 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:49.382191 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:49.443336 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.443594 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.443696 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.444406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.446491 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:49.483466 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 14:07:49.492985 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:49.540880 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.543511 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.544015 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.544204 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.545311 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.641714 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.642046 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.644481 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.646072 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.655001 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:49.674394 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:07:49.737205 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.739670 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.740556 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.740711 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.740908 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:49.807417 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:49.826843 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.832984 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.833459 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.833840 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.846240 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:49.938747 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:49.942914 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:49.948226 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:49.952443 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:49.955169 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:49.984022 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:50.031610 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:50.038496 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.038765 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.038795 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.048440 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.050298 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:50.080569 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:50.101403 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:50.130749 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.132050 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.132738 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.133198 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.134562 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.225632 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.227030 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.227188 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.228822 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.245700 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:50.248128 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:50.298885 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
I20240502 14:07:50.331455 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.333427 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.333441 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.335263 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.345949 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:50.369865 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:50.440380 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.442035 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.443727 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.452473 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.452826 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:50.469709 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:50.530429 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.531100 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.531266 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.534253 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.534440 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.620714 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.621183 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.622582 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.622954 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.623853 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.743685 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.746603 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.746949 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.748608 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.753196 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:50.807704 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:50.811723 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 14:07:50.833374 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.833817 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.836769 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:50.836835 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:50.837393 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:50.858251 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:50.929673 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:50.930312 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:50.937276 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:50.937409 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:50.941372 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:50.950479 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:50.958003 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:51.027058 29226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:07:51.027856 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
I20240502 14:07:51.028136 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.032707 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.034644 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.035171 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.036037 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:51.053830 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:51.127554 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:51.138458 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.142761 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.143484 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.144541 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.153779 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.228994 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.230211 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.231153 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.232908 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.245052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:51.270519 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 14:07:51.329331 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.330372 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.332119 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.332190 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.332861 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:51.364049 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:51.385064 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:51.420794 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.421788 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.422462 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.422631 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.423385 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.525081 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.525772 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.526995 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.527439 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.527485 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:51.555346 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:51.617867 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.619030 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.625202 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.625609 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.628137 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.714934 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.715165 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.718638 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.725626 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.727336 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:51.771908 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:51.808251 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:51.810496 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.812780 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.817775 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.820530 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.921654 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:51.923707 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:51.924556 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:51.932209 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:51.937728 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.015084 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.019201 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.019577 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.023563 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.023672 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:52.072376 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:52.095057 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240502 14:07:52.107442 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:52.110013 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:52.110502 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.119266 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.120111 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.121098 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.206338 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.207034 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.207435 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.208261 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.212787 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.313123 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.316535 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.317692 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.319298 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.322383 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.404237 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.410322 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.410719 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.413125 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.420809 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:52.466210 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:52.499794 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.499977 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.502038 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.504261 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.513770 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.593458 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.593497 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.594180 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.601929 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.605383 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:52.609459 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:52.659240 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:52.682075 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 14:07:52.682902 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240502 14:07:52.693352 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:52.704806 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.706207 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.706370 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.708093 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.725849 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.800614 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.804860 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.805223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:52.808373 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.821094 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:52.835111 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:52.848968 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 14:07:52.913980 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:52.914373 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:52.915414 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:52.918895 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:52.924443 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.011934 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.017753 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.018117 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.028409 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.030644 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.115492 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.115855 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.116106 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.116554 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.117141 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:53.127923 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:53.190821 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:53.201050 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.201846 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.203032 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.208842 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.216615 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.313975 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.316135 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.319326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.323100 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.328696 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:53.384124 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:53.410192 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.410391 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.412911 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.413115 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.421483 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.496290 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.501288 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.506886 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.507122 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.508369 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:53.542186 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240502 14:07:53.552649 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:53.594172 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.595225 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.595716 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.597509 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.598361 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:53.616817 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:53.667611 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 14:07:53.673128 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:53.711079 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.713093 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.714354 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.716048 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.717329 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:53.779520 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:53.810788 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.811422 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:53.814082 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.814555 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:53.832392 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:53.838650 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.906363 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:53.907347 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:53.917590 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:53.919101 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:53.920434 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.002948 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.005731 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.006225 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.008426 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.012583 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:54.052735 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:54.066550 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:07:54.117252 16835 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:07:54.117380 16835 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:07:54.117823 16835 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:07:54.118014 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:07:54.118151 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:07:54.118805 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:07:54.118919 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:07:54.118953 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:07:54.119093 16835 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:07:54.128731 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.128904 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.129267 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.129935 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.131433 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:54.151243 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:54.192824 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:54.216171 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.216580 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.216799 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.217427 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.222363 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.303316 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.306510 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.307906 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.308925 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.317880 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.391599 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.392129 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.393167 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.395529 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.406785 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:54.429651 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240502 14:07:54.505535 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:54.514549 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.519138 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.521520 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.521734 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.522457 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.610589 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.612943 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.615216 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.617008 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.626803 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:54.627796 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:54.646378 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240502 14:07:54.702350 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.702383 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.702662 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.703655 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.703953 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.795636 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.798414 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.803298 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.803647 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:54.813759 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.913146 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:54.913933 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:54.918506 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:54.919090 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:54.921890 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.012441 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.012768 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.013212 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.015537 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.020725 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:55.037922 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:55.053931 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:07:55.094969 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240502 14:07:55.107436 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.111454 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.112121 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.121642 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.125480 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.204655 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.205122 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.206683 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.208603 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.219944 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:55.244863 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240502 14:07:55.317188 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.318832 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.321728 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.322086 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.333278 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:55.367389 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:55.409833 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.413391 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.414484 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.415858 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.425848 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:55.434005 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:55.456185 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 14:07:55.501744 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.503994 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.505007 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.505833 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.509030 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:55.599186 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:55.599284 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.599992 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.601087 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.610489 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.610560 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:55.648527 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:55.709506 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.710359 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.710768 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.711211 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:55.725049 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:55.770218 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:55.809938 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.813007 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.813699 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.821643 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.828586 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:55.834276 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:07:55.906054 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:55.917491 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:55.918290 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:55.924430 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:55.925220 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:55.928117 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:56.011909 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:56.016379 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.017151 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.018389 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.019385 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.019963 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:56.063346 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 14:07:56.109637 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.113269 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.120420 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.122557 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.123034 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:56.135789 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:56.179471 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:56.206894 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.209192 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.209769 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.210640 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.211597 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.307730 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.308387 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.312285 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.313225 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.317935 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.400794 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.400871 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.403398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.405567 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:56.415858 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:56.422492 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:56.484167 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 14:07:56.493355 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:56.500182 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.502857 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.504508 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.506314 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:56.512729 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:56.513252 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:56.566977 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:56.595206 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.598968 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.606346 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.612176 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.615954 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:56.626003 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:56.700714 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.701294 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.703251 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.704406 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.714784 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.787190 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.789229 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.792956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:56.793414 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.804806 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.905011 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:56.909077 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:56.910116 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:56.910495 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:56.912647 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:56.945930 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:56.947340 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240502 14:07:57.000063 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.003072 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.004443 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.004621 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.012763 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:57.089067 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:57.091598 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.100414 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.102568 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.102807 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.113986 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.192253 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.192250 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.193581 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.194661 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.200451 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.297391 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.298409 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.299569 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.300026 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.300446 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:57.366230 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:57.390875 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.396356 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.400446 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.401654 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.406708 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:57.468119 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:57.486029 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.488267 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.490458 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.491832 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.492832 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.574223 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.579901 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.580670 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.584753 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.588413 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:57.673971 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:57.696770 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.698916 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.701956 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.702569 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.708608 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.787021 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.787591 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.793260 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.807981 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.808948 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:57.878859 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240502 14:07:57.879957 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:57.881378 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.882382 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.889590 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:57.890488 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 14:07:57.891395 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:57.893469 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:57.979743 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:57.979866 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:57.980845 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:57.981714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:57.983516 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:58.034965 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 14:07:58.045356 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:07:58.067709 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:58.092547 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.100452 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.102386 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.102605 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.109179 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.189021 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.190182 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.192962 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.193316 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.197451 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:58.205279 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:58.276377 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.278638 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.280422 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.281294 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.282549 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:58.292220 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:58.306753 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:58.367581 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.367851 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.369405 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.370240 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.391110 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.488359 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.488454 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.489539 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.491966 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.492555 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:58.509634 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:58.580807 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.583113 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.584343 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.588972 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.594678 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:58.645423 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:58.672861 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.674624 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.675374 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.676384 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.676977 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:58.744800 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:58.759541 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.764447 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.764662 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.764935 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.765955 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.884752 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:58.885363 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.886055 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:58.887814 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.890640 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:58.975116 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:58.980648 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:58.980688 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 14:07:58.984326 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:58.986879 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:58.992589 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:58.999374 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:07:59.043200 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:07:59.074944 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.077968 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.078060 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.080435 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.082370 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:59.100248 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:07:59.125699 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 14:07:59.126504 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:59.166149 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.169697 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.170665 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.171329 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.180949 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:59.189615 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:59.293839 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.295802 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.296562 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.301926 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.303189 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.389873 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.391333 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.391638 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:07:59.393348 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:07:59.394505 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.396677 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.480002 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.481700 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.483009 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.483197 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.493665 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:59.508780 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:07:59.528302 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:59.569416 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.569873 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.571149 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.571241 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.573331 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:07:59.646710 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:07:59.647388 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:07:59.689633 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.690619 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.690704 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.693387 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.709466 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:07:59.722079 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:07:59.780784 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.780824 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.789036 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.794625 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.804576 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.872547 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.873204 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.875406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.877274 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:07:59.882882 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:07:59.961141 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:07:59.964962 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:07:59.970563 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:07:59.970793 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:07:59.977716 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:07:59.979053 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.087713 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.092598 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.093329 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.100406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.101696 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.183907 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.187997 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.188099 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.196319 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.202071 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.275795 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.276648 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.277418 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.278673 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.279826 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:00.297562 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:08:00.348788 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:00.361591 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.364442 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.365274 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.365901 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.367307 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:00.376875 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240502 14:08:00.410665 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:00.484445 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.484730 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.488445 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.492009 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.494410 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:00.571910 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:08:00.580673 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.582690 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.583761 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.584628 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.592186 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:00.627104 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:00.667691 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.670545 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.674237 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.674566 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.688621 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.769263 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.770553 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.771680 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.778575 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.789242 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:08:00.829007 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 14:08:00.845484 28961 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 14:08:00.854835 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:00.880422 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:00.880553 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.883277 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.883563 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.884414 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:00.909391 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:08:00.950752 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:00.979221 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:00.979331 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:00.980134 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:00.982872 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:00.983508 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.077270 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.077697 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.078505 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.079730 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.081090 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:01.087607 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 14:08:01.147411 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:01.170642 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.173285 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.173959 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.179865 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.189983 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.280918 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.280964 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.283252 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.283711 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.293087 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.368027 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.376423 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.384922 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.386447 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.393494 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:01.396826 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:01.471457 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.474496 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.474536 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.474913 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.478371 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:01.508687 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:08:01.550453 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:01.563580 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.566782 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.566954 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.567253 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.578274 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:01.641696 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 14:08:01.681828 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.682056 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.683687 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.687013 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.693320 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:01.702893 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:08:01.759284 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:01.771081 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:08:01.774648 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:01.775907 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.776736 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.782150 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.794355 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:01.848001 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:08:01.868729 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.872762 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.875630 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.883529 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:01.893299 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.973660 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:01.973932 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:01.974020 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:01.975191 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:01.975378 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.075044 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.075093 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.082870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.085177 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.104163 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:02.127849 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:02.178324 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.178561 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.180459 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.181155 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.182507 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:02.222322 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:02.271371 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.273620 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.274825 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.281456 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.282743 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:02.333437 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:08:02.358453 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.363027 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.366291 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.367777 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.372035 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.474745 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.477301 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.477350 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.479130 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.488603 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.567667 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.572044 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.572979 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.574301 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.590436 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:08:02.615664 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:02.660531 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.664681 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.666460 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.667340 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.674095 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.758297 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.761382 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.762152 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.763263 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.769523 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:02.799960 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:02.874143 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:08:02.877668 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:02.878401 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:02.879256 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.882647 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.883241 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:02.928221 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240502 14:08:02.953629 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
I20240502 14:08:02.978785 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:02.982969 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:02.985862 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:02.986030 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:02.987501 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.072369 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.073927 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.078285 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:03.078920 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.093170 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:03.099848 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:03.168704 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.171227 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.171563 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.174047 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:03.176908 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:03.188150 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 14:08:03.217336 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:03.289901 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.290661 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.290889 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.291615 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.304473 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:03.338361 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:03.380759 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.386340 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.388485 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:03.397367 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.403187 20470 raft_consensus.cc:482] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 22dbbf207ba0440392e71a10c2ca78ca)
I20240502 14:08:03.403303 20470 raft_consensus.cc:504] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd692a8958c241fca3aec5cca3a0ded5" member_type: VOTER last_known_addr { host: "127.16.217.3" port: 41345 } } peers { permanent_uuid: "a7221b90d0aa4770a8ce5522103a96f4" member_type: VOTER last_known_addr { host: "127.16.217.1" port: 37835 } } peers { permanent_uuid: "22dbbf207ba0440392e71a10c2ca78ca" member_type: VOTER last_known_addr { host: "127.16.217.5" port: 45769 } } peers { permanent_uuid: "33ebf308f6f24e6e8768ef5f14ddc785" member_type: VOTER last_known_addr { host: "127.16.217.7" port: 40969 } }
I20240502 14:08:03.403856 20470 leader_election.cc:288] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a7221b90d0aa4770a8ce5522103a96f4 (127.16.217.1:37835), 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769), 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969)
I20240502 14:08:03.405401 29027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "372e9cc5aaed4b80b3b451efdb3e94fc" candidate_uuid: "cd692a8958c241fca3aec5cca3a0ded5" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "a7221b90d0aa4770a8ce5522103a96f4" is_pre_election: true
I20240502 14:08:03.405563 29027 raft_consensus.cc:2454] T 372e9cc5aaed4b80b3b451efdb3e94fc P a7221b90d0aa4770a8ce5522103a96f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd692a8958c241fca3aec5cca3a0ded5 in term 2.
W20240502 14:08:03.406411 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111)
W20240502 14:08:03.406529 29226 leader_election.cc:334] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111)
I20240502 14:08:03.406576 29226 leader_election.cc:302] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: a7221b90d0aa4770a8ce5522103a96f4, cd692a8958c241fca3aec5cca3a0ded5; no voters: 22dbbf207ba0440392e71a10c2ca78ca, 33ebf308f6f24e6e8768ef5f14ddc785
I20240502 14:08:03.406786 20470 raft_consensus.cc:2726] T 372e9cc5aaed4b80b3b451efdb3e94fc P cd692a8958c241fca3aec5cca3a0ded5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:08:03.407058 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.474485 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.475423 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.477044 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.477296 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.481247 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:03.486343 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 14:08:03.499162 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:03.566474 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.567067 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.567955 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.569391 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.570318 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:03.581533 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:08:03.605540 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:03.687108 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.688185 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.690140 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.697165 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.697492 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:03.767118 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:03.778138 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.779739 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.782094 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.782254 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.791536 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:03.873859 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.874131 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.876499 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.880465 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.881744 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:03.973601 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:03.974004 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:03.974876 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:03.978049 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:03.993291 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:04.021484 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 14:08:04.078637 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.080724 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.082805 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.088563 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.089447 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:04.121085 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:08:04.154284 29093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20240502 14:08:04.172128 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.172892 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.173668 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.174504 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.190223 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.268415 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.273321 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.273695 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.274061 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.274340 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:04.276749 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:08:04.285488 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:08:04.307770 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:08:04.322346 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:08:04.360042 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.361233 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.361392 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.362741 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.375463 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:04.443581 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:04.473037 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.473661 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.475628 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.476621 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.476989 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.559854 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.562482 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.565471 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.568419 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.568791 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.659421 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.660543 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.662426 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.664673 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.673055 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:04.712805 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:08:04.723181 29357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240502 14:08:04.723906 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:04.748224 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.749781 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.750128 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.750762 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.757915 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:04.862533 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.865327 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.867456 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.867556 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:04.882280 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:08:04.884734 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.957943 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:04.959205 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:04.959527 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:04.970084 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:04.970898 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:05.027587 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:05.052183 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.052233 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.060446 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.062628 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.063408 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:05.099545 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:05.143316 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.144212 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.145339 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.145893 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.147749 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.251619 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.256083 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.256480 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.260507 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.262883 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.346727 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.346817 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:05.348301 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:05.350958 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.351398 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.362978 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.442180 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.442837 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.444778 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.446218 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.446645 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:05.486322 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240502 14:08:05.487538 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
I20240502 14:08:05.536053 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.538376 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.538528 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.539477 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.548169 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:05.645074 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:05.651686 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.652307 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.653553 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.654955 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.668870 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:05.741940 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.743898 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.744243 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.746448 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.746708 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:05.807765 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:05.843806 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.847535 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.848083 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.865411 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.866633 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:05.883929 28960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 14:08:05.922168 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:05.943269 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:05.945717 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:05.954243 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:05.954883 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:05.961102 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:05.969583 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 14:08:05.987082 31115 consensus_peers.cc:485] T cfd44eeeec934413a5001673430e6500 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:08:06.006989 29225 consensus_peers.cc:485] T ff65ce0568514e89894ec845fbda31f3 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:06.049227 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.049286 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.054138 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.054947 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.056731 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:06.093041 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 14:08:06.118253 29225 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 14:08:06.142645 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.143779 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.144052 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.145658 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.153771 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:06.170574 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:06.245023 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.245554 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.249547 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.258355 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.258878 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.339798 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.340193 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.341272 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.344200 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.349274 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.441462 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.441890 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.443703 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.448830 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.450948 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.533860 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.536777 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.537778 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.549333 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.557037 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:06.582509 29225 consensus_peers.cc:485] T a75497cf22014c0cb9d079b448284b3d P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:08:06.611191 29357 consensus_peers.cc:485] T 0176fb1a25a94c1c9d8cf0dc63577e91 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 14:08:06.636185 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.638178 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.639477 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.644399 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.654353 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:06.710031 28961 consensus_peers.cc:485] T db68071f838a4e209f3db69bea6e54f3 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:06.727156 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.729106 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.729750 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.737267 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.738406 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:06.788627 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
W20240502 14:08:06.823833 28960 consensus_peers.cc:485] T 81b2d5d0f4044106bed079c3faa72447 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:06.837491 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.840574 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:06.851501 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.852375 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.861649 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:06.906769 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
I20240502 14:08:06.939860 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:06.942588 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:06.942885 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:06.943020 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:06.943534 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:08:06.965366 29225 consensus_peers.cc:485] T 5e1b37b67e104154b6755c1762f8f6ff P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:07.031198 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.032915 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.033511 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.038482 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.052498 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.125912 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.127205 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.128729 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.129907 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.144470 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:07.181340 29357 consensus_peers.cc:485] T 015b587864d54da8aee36c3afd968e86 P edce308627b14fa6968d90ec77d51aec -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:07.251433 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.252327 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.253804 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.257441 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.270920 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.346472 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.347433 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.351398 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.353163 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.362783 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 14:08:07.400264 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:07.439311 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.440361 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.442147 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.444191 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.464257 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.535444 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.537122 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.540714 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.541563 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.545471 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:07.634914 28960 consensus_peers.cc:485] T c88f85f143d845a1b7066244c606f8ba P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:07.649618 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.652801 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.652808 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.654284 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.654608 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:07.663681 28961 consensus_peers.cc:485] T 19784717201a464c84607d2a43e651c2 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
I20240502 14:08:07.740760 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.741411 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.744197 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.748078 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.748447 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 14:08:07.757252 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 22dbbf207ba0440392e71a10c2ca78ca (127.16.217.5:45769): Couldn't send request to peer 22dbbf207ba0440392e71a10c2ca78ca. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.5:45769: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240502 14:08:07.831316 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.833917 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.836233 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.836589 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.858328 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:07.935191 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:07.940488 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:07.940680 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:07.941134 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:07.956720 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:08.010450 29226 consensus_peers.cc:485] T 2b1adebdbd104796b927963e86a15295 P cd692a8958c241fca3aec5cca3a0ded5 -> Peer 33ebf308f6f24e6e8768ef5f14ddc785 (127.16.217.7:40969): Couldn't send request to peer 33ebf308f6f24e6e8768ef5f14ddc785. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.7:40969: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240502 14:08:08.021020 29093 consensus_peers.cc:462] T ed713f89a4fd47d79f03711dd14c1334 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111)
I20240502 14:08:08.042600 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.043790 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.044239 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:08.045979 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 14:08:08.056581 28960 consensus_peers.cc:485] T 5a91fd318f0844c1961f16823a7ccc88 P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:08.056838 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:08.066345 31115 consensus_peers.cc:485] T d5ffcf19e66e4adca28d87cbe2245964 P c59dc7abff9749abab221ce409d49f59 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 14:08:08.144476 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.144730 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:08.145058 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.146390 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:08.160394 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:08.245564 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:08.248991 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.249222 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.253682 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:08.259742 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:08.339458 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.340939 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:08.340951 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.343317 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:08:08.349473 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 14:08:08.389588 28960 consensus_peers.cc:485] T 785828dee4594ca8813b9418cc5dba2d P a7221b90d0aa4770a8ce5522103a96f4 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 14:08:08.444262 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:08.445037 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:08.447280 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.450294 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.452781 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:08.461490 29093 consensus_peers.cc:485] T 4aa944cd0b82429da02bd3dd35e0cce7 P bd40abfe1fbd48529f9ebbcc4d3abf86 -> Peer 12348ca540074b32912fdd6333484e0b (127.16.217.6:38737): Couldn't send request to peer 12348ca540074b32912fdd6333484e0b. Status: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 14:08:08.494130 31115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.217.6:38737: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 14:08:08.539978 29141 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:08:08.544149 29403 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:08:08.545063 29006 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:08:08.545902 31161 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:08:08.548408 29273 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 14:08:08.552332 29357 consensus_peers.cc:485] T a476d1664de7456ab21ddbf2e12b4f0d P edce308627b14fa6968d90ec77d51aec ->